idnits 2.17.1 draft-zhou-netconf-multi-stream-originators-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 18, 2017) is 2375 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-14) exists of draft-ietf-core-coap-pubsub-02 == Outdated reference: A later version (-25) exists of draft-ietf-netconf-yang-push-10 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group T. Zhou 3 Internet-Draft G. Zheng 4 Intended status: Standards Track Huawei 5 Expires: April 21, 2018 E. Voit 6 Cisco Systems 7 A. Clemm 8 Huawei 9 A. Bierman 10 YumaWorks 11 October 18, 2017 13 Subscription to Multiple Stream Originators 14 draft-zhou-netconf-multi-stream-originators-00 16 Abstract 18 This document describes the distributed data collection mechanism 19 that allows multiple data streams to be managed using a single 20 subscription. Specifically, multiple data streams are pushed 21 directly to the collector without passing through a broker for 22 internal consolidation. 24 Requirements Language 26 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 27 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 28 document are to be interpreted as described in RFC 2119 [RFC2119]. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on April 21, 2018. 47 Copyright Notice 49 Copyright (c) 2017 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 65 2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 3 66 2.1. Use Case 1: Data Collection from Devices with Main-board 67 and Line-cards . . . . . . . . . . . . . . . . . . . . . 3 68 2.2. Use Case 2: IoT Data Collection . . . . . . . . . . . . . 4 69 3. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 5 70 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 71 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 72 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 73 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 74 7.1. Normative References . . . . . . . . . . . . . . . . . . 7 75 7.2. Informative References . . . . . . . . . . . . . . . . . 8 76 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 8 77 Appendix B. Subscription Decomposition . . . . . . . . . . . . . 8 78 Appendix C. Publication Composition . . . . . . . . . . . . . . 9 79 Appendix D. Examples . . . . . . . . . . . . . . . . . . . . . . 10 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 82 1. Introduction 84 Streaming telemetry refers to sending a continuous stream of 85 operational data from a device to a remote receiver. This provides 86 an ability to monitor a network from remote and to provide network 87 analytics. Devices generate telemetry data and push that data to a 88 collector for further analysis. By streaming the data, much better 89 performance, finer-grained sampling, monitoring accuracy, and 90 bandwidth utilization can be achieved than with polling-based 91 alternatives. 93 YANG-Push [I-D.ietf-netconf-yang-push] defines a transport- 94 independent subscription mechanism for datastore updates, in which a 95 subscriber can subscribe to a stream of datastore updates from a 96 server, or update provider. The current design involves subscription 97 to a single push server. This conceptually centralized model 98 encounters efficiency limitations in cases where the data sources are 99 themselves distributed, such as line cards in a piece of network 100 equipment. In such cases, it will be a lot more efficient to have 101 each data source (e.g., each line card) originate its own stream of 102 updates, rather than requiring updates to be tunneled through a 103 central server where they are combined. What is needed is a 104 distributed mechanism that allows to directly push multiple 105 individual data substreams, without needing to first pass them 106 through an additional processing stage for internal consolidation, 107 but still allowing those substreams to be managed and controlled via 108 a single subscription. 110 This document will describe such distributed data collection 111 mechanism and how it can work by extending existing YANG-Push 112 mechanism. The proposal is general enough to fit many scenarios. 114 2. Use Cases 116 2.1. Use Case 1: Data Collection from Devices with Main-board and Line- 117 cards 119 For data collection from devices with main-board and line-cards, 120 existing YANG-Push solutions consider only one push server typically 121 reside in the main board. As shown in the following figure, data are 122 collected from line cards and aggregate to the main board as one 123 consolidated stream. So the main board can easily become the 124 performance bottle-neck. The optimization is to apply the 125 distributed data collection mechanism which can directly push data 126 from line cards to a collector. On one hand, this will reduce the 127 cost of scarce compute and memory resources on the main board for 128 data processing and assembling. On the other hand, distributed data 129 push can off-load the streaming traffic to multiple interfaces. 131 +-------------------------------------+ 132 | collector | 133 +------^-----------^-----------^------+ 134 | | | 135 | | | 136 +-------------------------------------+ 137 | | | | | 138 | | +-----+------+ | | 139 | | | main board | | | 140 | | +--^-----^---+ | | 141 | | | | | | 142 | | +---+ +---+ | | 143 | | | | | | 144 | +----+----+---+ +---+----+----+ | 145 | | line card 1 | | line card 2 | | 146 | +-------------+ +-------------+ | 147 | device | 148 +-------------------------------------+ 150 Fig. 1 Data Collection from Devices with Main-board and Line-cards 152 2.2. Use Case 2: IoT Data Collection 154 In the IoT data collection scenario, as shown in the following 155 figure, collector usually cannot access to IoT nodes directly, but is 156 isolated by the border router. So the collector subscribes data from 157 the border router, and let the border router to disassemble the 158 subscription to corresponding IoT nodes. The border router is 159 typically the traffic convergence point. It's intuitive to treat the 160 border router as a broker assembling the data collected from the IoT 161 nodes and forwarding to the collector[I-D.ietf-core-coap-pubsub]. 162 However, the border router is not so powerful on data assembling as a 163 network device. It's more efficient for the collector, which may be 164 a server or even a cluster, to assemble the subscribed data if 165 possible. In this case, push servers that reside in IoT nodes can 166 stream data to the collector directly while traffic only passes 167 through the border router. 169 +-------------------------------+ 170 | collector | 171 +---^-----------^------------^--+ 172 | | | 173 | | | 174 | | | 175 | +-------+--------+ | 176 | | border router | | 177 | +----^------^----+ | 178 | | | | 179 | | | | 180 | +---+ +---+ | 181 | | | | 182 +---+----+---+ +---+----+---+ 183 | IoT node 1 | | IoT node 2 | 184 +------------+ +------------+ 186 Fig. 2 IoT Data Collection 188 3. Solution Overview 190 All the use cases described in the previous section are very similar 191 on the data subscription and publication mode, hence can be 192 abstracted to the following generic distributed data collection 193 framework, as shown in the following figure. 195 A Collector usually includes two components, 197 o the Subscriber generates the subscription instructions to express 198 what and how the collector want to receive the data; 200 o the Receiver is the target for the data publication. 202 For one subscription, there may be one to many receivers. And the 203 subscriber does not necessarily share the same address with 204 receivers. 206 In this framework, the stream originators have the Master role and 207 the Agent role. Both the Master and the Agent include two 208 components, 210 o the Subscription Server manages capabilities that it can provide 211 to the subscriber. 213 o the Publication Server pushes data to the receiver according to 214 the subscription information. 216 The Master knows all the capabilities that the attached Agents and 217 itself can provide, and exposes the global capability to the 218 Collector. The Collector cannot see the Agents directly, so it will 219 only send the subscription information to the Master. The Master 220 disassembles the subscription to multiple component subscriptions, 221 each involving data from a separate telemetry source. The component 222 subscriptions are then distributed to the corresponding Agents. 224 When data streaming, the Publication Server located in each stream 225 originator collects and encapsulates the packets per the component 226 subscription, and pushes the piece of data which it can serve 227 directly to the designated data Collector. The Collector is able to 228 assemble many pieces of data associated with one subscription, and 229 can also deduce the missing pieces of data. 231 +-------------------------------------+ 232 | Collector | 233 | +------------+ +------------+ | 234 | | Subscriber | | Receiver <-------+ 235 | +-^----+-----+ +------^-----+ | | 236 | | | | | | 237 +-------------------------------------+ | 238 capability | |subscription | push | 239 | | | | 240 +-------------------------------------+ | 241 | | | Master | | | 242 | +--+----v------+ +------+------+ | | 243 | | Subscription | | Publication | | | 244 | | Server | | Server | | | 245 | +--^----+------+ +-------------+ | | 246 | | | | | 247 +-------------------------------------+ | 248 component | | component push | 249 capability | | subscription | 250 +-------------------------------------+ | 251 | | | Agent | | 252 | +--+----v------+ +-------------+ | | 253 | | Component | | Publication | | | 254 | | Subscription | | Server +------+ 255 | | Server | +-------------+ | 256 | +--------------+ | 257 +-------------------------------------+ 259 Fig. 3 The Generic Distributed Data Collection Framework 261 Master and Agents may interact with each other in several ways: 263 o Agents need to have a registration or announcement handshake with 264 the Master Node, so the Master Node is aware of them and of 265 lifecycle events (such as Agent appearing and disappearing). 267 o The Master relays the component subscriptions to the Agents. 269 o The Agents indicate status of component subscriptions to the 270 Master. The status of the overall subscription is maintained by 271 the Master. The Master is also responsible for notifying the 272 subscriber in case of any problems of component subscriptions. 274 The details of the interaction between the Master and the Agent is 275 out of scope of this document. It may be described by a dedicated 276 protocol specification. 278 During the subscription and the associated publication process, this 279 document assumes all the Agents registered to the Mater can always 280 provide the announced capabilities. 282 Note: Some preliminary considerations on the solution details are now 283 listed in the appendix for reference. The detailed solution need to 284 be discussed and will be added if the WG accepts the problem 285 statement. 287 4. IANA Considerations 289 This document makes no request of IANA. 291 Note to RFC Editor: this section may be removed on publication as an 292 RFC. 294 5. Security Considerations 296 6. Acknowledgements 298 7. References 300 7.1. Normative References 302 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 303 Requirement Levels", BCP 14, RFC 2119, 304 DOI 10.17487/RFC2119, March 1997, 305 . 307 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 308 and A. Bierman, Ed., "Network Configuration Protocol 309 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 310 . 312 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 313 RFC 7950, DOI 10.17487/RFC7950, August 2016, 314 . 316 [W3C.REC-xpath-19991116] 317 Clark, J. and S. DeRose, "XML Path Language (XPath) 318 Version 1.0", World Wide Web Consortium Recommendation 319 REC-xpath-19991116, November 1999, 320 . 322 7.2. Informative References 324 [I-D.ietf-core-coap-pubsub] 325 Koster, M., Keranen, A., and J. Jimenez, "Publish- 326 Subscribe Broker for the Constrained Application Protocol 327 (CoAP)", draft-ietf-core-coap-pubsub-02 (work in 328 progress), July 2017. 330 [I-D.ietf-netconf-yang-push] 331 Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen- 332 Nygaard, E., Bierman, A., and B. Lengyel, "Subscribing to 333 YANG datastore push updates", draft-ietf-netconf-yang- 334 push-10 (work in progress), October 2017. 336 Appendix A. Change Log 338 (To be removed by RFC editor prior to publication) 340 Appendix B. Subscription Decomposition 342 Since Agents are invisible to the Collector, the Collector can only 343 subscribe to the Master. This requires the Master to: 345 1. expose the global capability that can be served by multiple 346 stream originators; 348 2. disassemble the subscription to multiple component subscriptions, 349 and distribute them to the corresponding telemetry sources. 351 To achieve the above requirement, the Master need a global capability 352 description which is typically the YANG [RFC7950] data model. This 353 global YANG model is provided as the contract between the Master and 354 the Collector. 356 The Master also need a data structure, typically a table as shown 357 below, to keep track the mapping between the resource and the 358 corresponding location identifier of the node that commits to serve 359 the data. For the YANG defined capabilities, the resource is 360 described using the XPath [W3C.REC-xpath-19991116] expression. 362 +------------+---------------+ 363 | resource | location ID | 364 +------------+---------------+ 366 Table 1 368 Each Agent associating to the Master owns a local YANG model to 369 describe the capabilities which it can serve as part of the global 370 capability. All the Agents need to know the namespace associate with 371 the Master. 373 YANG-Push supports two filtering syntaxes which are XPath and Subtree 374 [RFC6241]. When a YANG-Push subscription request arrives, the Master 375 Node will firstly extract the filter information. Consequently, 376 according to the resource-locationID table, the master subscription 377 can be disassembled into multiple component subscriptions, and the 378 corresponding location ID can be associated. The component 379 subscriptions share the same Subscription ID as the master 380 subscription. 382 Appendix C. Publication Composition 384 The Publication Server collects and encapsulates the packets per the 385 component subscription. There are several potential encodings, 386 including XML, JSON, CBOR and GPB. The encoding of the data records 387 follows the YANG schema, so that the composition at the Receiver can 388 benefit from the structured and hierarchical data instance. The 389 Collector may be able to assemble many pieces of data associated with 390 one subscription, and can also deduce the missing pieces of data. 392 The Collector recognizes data records associated with one 393 subscription according the Subscription ID. Data records generated 394 per one subscription are assigned with the same Subscription ID. 396 For the time series data stream, records are produced periodically 397 from each stream originator. The message arrival time varies because 398 of the distributed nature of the publication. The receiver assembles 399 data generated at the same time period based on the recording time 400 consisted in each data record. In this case, time synchronization is 401 required for all the steam originators. 403 Appendix D. Examples 405 TBD 407 Authors' Addresses 409 Tianran Zhou 410 Huawei 411 156 Beiqing Rd., Haidian District 412 Beijing 413 China 415 Email: zhoutianran@huawei.com 417 Guangying Zheng 418 Huawei 419 101 Yu-Hua-Tai Software Road 420 Nanjing, Jiangsu 421 China 423 Email: zhengguangying@huawei.com 425 Eric Voit 426 Cisco Systems 427 United States of America 429 Email: evoit@cisco.com 431 Alexander Clemm 432 Huawei 433 2330 Central Expressway 434 Santa Clara, California 435 United States of America 437 Email: alexander.clemm@huawei.com 439 Andy Bierman 440 YumaWorks 441 United States of America 443 Email: andy@yumaworks.com