idnits 2.17.1 draft-zhou-netconf-multi-stream-originators-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 11, 2017) is 2348 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 5246 (Obsoleted by RFC 8446) ** Obsolete normative reference: RFC 6347 (Obsoleted by RFC 9147) == Outdated reference: A later version (-13) exists of draft-ietf-core-coap-pubsub-02 == Outdated reference: A later version (-25) exists of draft-ietf-netconf-yang-push-11 Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NETCONF T. Zhou 3 Internet-Draft G. Zheng 4 Intended status: Standards Track Huawei 5 Expires: May 15, 2018 E. Voit 6 Cisco Systems 7 A. Clemm 8 Huawei 9 A. Bierman 10 YumaWorks 11 November 11, 2017 13 Subscription to Multiple Stream Originators 14 draft-zhou-netconf-multi-stream-originators-01 16 Abstract 18 This document describes the distributed data collection mechanism 19 that allows multiple data streams to be managed using a single 20 subscription. Specifically, multiple data streams are pushed 21 directly to the collector without passing through a broker for 22 internal consolidation. 24 Requirements Language 26 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 27 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 28 document are to be interpreted as described in RFC 2119 [RFC2119]. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on May 15, 2018. 47 Copyright Notice 49 Copyright (c) 2017 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 65 2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 3 66 2.1. Use Case 1: Data Collection from Devices with Main-board 67 and Line-cards . . . . . . . . . . . . . . . . . . . . . 3 68 2.2. Use Case 2: IoT Data Collection . . . . . . . . . . . . . 4 69 3. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 5 70 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 71 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 72 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 73 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 74 7.1. Normative References . . . . . . . . . . . . . . . . . . 7 75 7.2. Informative References . . . . . . . . . . . . . . . . . 8 76 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 8 77 Appendix B. Subscription Decomposition . . . . . . . . . . . . . 8 78 Appendix C. Publication Composition . . . . . . . . . . . . . . 10 79 Appendix D. Subscription Management . . . . . . . . . . . . . . 10 80 Appendix E. Notifications on Subscription State Changes . . . . 11 81 Appendix F. Configured Subscription and Call Home . . . . . . . 11 82 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 84 1. Introduction 86 Streaming telemetry refers to sending a continuous stream of 87 operational data from a device to a remote receiver. This provides 88 an ability to monitor a network from remote and to provide network 89 analytics. Devices generate telemetry data and push that data to a 90 collector for further analysis. By streaming the data, much better 91 performance, finer-grained sampling, monitoring accuracy, and 92 bandwidth utilization can be achieved than with polling-based 93 alternatives. 95 YANG-Push [I-D.ietf-netconf-yang-push] defines a transport- 96 independent subscription mechanism for datastore updates, in which a 97 subscriber can subscribe to a stream of datastore updates from a 98 server, or update provider. The current design involves subscription 99 to a single push server. This conceptually centralized model 100 encounters efficiency limitations in cases where the data sources are 101 themselves distributed, such as line cards in a piece of network 102 equipment. In such cases, it will be a lot more efficient to have 103 each data source (e.g., each line card) originate its own stream of 104 updates, rather than requiring updates to be tunneled through a 105 central server where they are combined. What is needed is a 106 distributed mechanism that allows to directly push multiple 107 individual data substreams, without needing to first pass them 108 through an additional processing stage for internal consolidation, 109 but still allowing those substreams to be managed and controlled via 110 a single subscription. 112 This document will describe such distributed data collection 113 mechanism and how it can work by extending existing YANG-Push 114 mechanism. The proposal is general enough to fit many scenarios. 116 2. Use Cases 118 2.1. Use Case 1: Data Collection from Devices with Main-board and Line- 119 cards 121 For data collection from devices with main-board and line-cards, 122 existing YANG-Push solutions consider only one push server typically 123 reside in the main board. As shown in the following figure, data are 124 collected from line cards and aggregate to the main board as one 125 consolidated stream. So the main board can easily become the 126 performance bottle-neck. The optimization is to apply the 127 distributed data collection mechanism which can directly push data 128 from line cards to a collector. On one hand, this will reduce the 129 cost of scarce compute and memory resources on the main board for 130 data processing and assembling. On the other hand, distributed data 131 push can off-load the streaming traffic to multiple interfaces. 133 +-------------------------------------+ 134 | collector | 135 +------^-----------^-----------^------+ 136 | | | 137 | | | 138 +-------------------------------------+ 139 | | | | | 140 | | +-----+------+ | | 141 | | | main board | | | 142 | | +--^-----^---+ | | 143 | | | | | | 144 | | +---+ +---+ | | 145 | | | | | | 146 | +----+----+---+ +---+----+----+ | 147 | | line card 1 | | line card 2 | | 148 | +-------------+ +-------------+ | 149 | device | 150 +-------------------------------------+ 152 Fig. 1 Data Collection from Devices with Main-board and Line-cards 154 2.2. Use Case 2: IoT Data Collection 156 In the IoT data collection scenario, as shown in the following 157 figure, collector usually cannot access to IoT nodes directly, but is 158 isolated by the border router. So the collector subscribes data from 159 the border router, and let the border router to disassemble the 160 subscription to corresponding IoT nodes. The border router is 161 typically the traffic convergence point. It's intuitive to treat the 162 border router as a broker assembling the data collected from the IoT 163 nodes and forwarding to the collector[I-D.ietf-core-coap-pubsub]. 164 However, the border router is not so powerful on data assembling as a 165 network device. It's more efficient for the collector, which may be 166 a server or even a cluster, to assemble the subscribed data if 167 possible. In this case, push servers that reside in IoT nodes can 168 stream data to the collector directly while traffic only passes 169 through the border router. 171 +-------------------------------+ 172 | collector | 173 +---^-----------^------------^--+ 174 | | | 175 | | | 176 | | | 177 | +-------+--------+ | 178 | | border router | | 179 | +----^------^----+ | 180 | | | | 181 | | | | 182 | +---+ +---+ | 183 | | | | 184 +---+----+---+ +---+----+---+ 185 | IoT node 1 | | IoT node 2 | 186 +------------+ +------------+ 188 Fig. 2 IoT Data Collection 190 3. Solution Overview 192 All the use cases described in the previous section are very similar 193 on the data subscription and publication mode, hence can be 194 abstracted to the following generic distributed data collection 195 framework, as shown in the following figure. 197 A Collector usually includes two components, 199 o the Subscriber generates the subscription instructions to express 200 what and how the collector want to receive the data; 202 o the Receiver is the target for the data publication. 204 For one subscription, there may be one to many receivers. And the 205 subscriber does not necessarily share the same address with 206 receivers. 208 In this framework, the stream originators have the Master role and 209 the Agent role. Both the Master and the Agent include two 210 components, 212 o the Subscription Server manages capabilities that it can provide 213 to the subscriber. 215 o the Publisher pushes data to the receiver according to the 216 subscription information. 218 The Master knows all the capabilities that the attached Agents and 219 itself can provide, and exposes the Global Capability to the 220 Collector. The Collector cannot see the Agents directly, so it will 221 only send the Global Subscription information to the Master. The 222 Master disassembles the Global Subscription to multiple Component 223 Subscriptions, each involving data from a separate telemetry source. 224 The Component Subscriptions are then distributed to the corresponding 225 Agents. 227 When data streaming, the Publisher located in each stream originator 228 collects and encapsulates the packets per the Component Subscription, 229 and pushes the piece of data which it can serve directly to the 230 designated data Collector. The Collector is able to assemble many 231 pieces of data associated with one Global Subscription, and can also 232 deduce the missing pieces of data. 234 +-------------------------------------+ 235 | Collector | 236 | +------------+ +------------+ | 237 | | Subscriber | | Receiver <-------+ 238 | +-^----+-----+ +------^-----+ | | 239 | | | | | | 240 +-------------------------------------+ | 241 Global | |Global | push | 242 Capability | |Subscription | | 243 +-------------------------------------+ | 244 | | | Master | | | 245 | +--+----v------+ +------+------+ | | 246 | | Subscription | | Publisher | | | 247 | | Server | | | | | 248 | +--^----+------+ +-------------+ | | 249 | | | | | 250 +-------------------------------------+ | 251 Component | | Component push | 252 Capability | | Subscription | 253 +-------------------------------------+ | 254 | | | Agent | | 255 | +--+----v------+ +-------------+ | | 256 | | Component | | Publisher | | | 257 | | Subscription | | +------+ 258 | | Server | +-------------+ | 259 | +--------------+ | 260 +-------------------------------------+ 262 Fig. 3 The Generic Distributed Data Collection Framework 264 Master and Agents may interact with each other in several ways: 266 o Agents need to have a registration or announcement handshake with 267 the Master, so the Master is aware of them and of life-cycle 268 events (such as Agent appearing and disappearing). 270 o The Master relays the component subscriptions to the Agents. 272 o The Agents indicate status of Component Subscriptions to the 273 Master. The status of the overall subscription is maintained by 274 the Master. The Master is also responsible for notifying the 275 subscriber in case of any problems of Component Subscriptions. 277 Any technical mechanisms or protocols used for the coordination of 278 operational information between Master and Agent is out-of-scope of 279 the solution. We will need to instrument the results of this 280 coordination on the Master Node. 282 Note: Some preliminary considerations on the solution details are now 283 listed in the appendix for reference. The detailed solution need to 284 be discussed and will be added if the WG accepts the problem 285 statement. 287 4. IANA Considerations 289 This document makes no request of IANA. 291 Note to RFC Editor: this section may be removed on publication as an 292 RFC. 294 5. Security Considerations 296 It's expected to reuse the existing secure transport layer protocols, 297 such as TLS [RFC5246] and DTLS [RFC6347], to secure the telemetry 298 stream. The Collector cannot access the Agent directly but to 299 negotiate the security parameters with the Master. However the data 300 streams are actually generated by the Agents which are invisible to 301 the Collector. So mechanisms may need to consider when adapting 302 secure transport layer protocols here. the detailed solution is TBD. 304 6. Acknowledgements 306 7. References 308 7.1. Normative References 310 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 311 Requirement Levels", BCP 14, RFC 2119, 312 DOI 10.17487/RFC2119, March 1997, 313 . 315 [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security 316 (TLS) Protocol Version 1.2", RFC 5246, 317 DOI 10.17487/RFC5246, August 2008, 318 . 320 [RFC6347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer 321 Security Version 1.2", RFC 6347, DOI 10.17487/RFC6347, 322 January 2012, . 324 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 325 RFC 7950, DOI 10.17487/RFC7950, August 2016, 326 . 328 7.2. Informative References 330 [I-D.ietf-core-coap-pubsub] 331 Koster, M., Keranen, A., and J. Jimenez, "Publish- 332 Subscribe Broker for the Constrained Application Protocol 333 (CoAP)", draft-ietf-core-coap-pubsub-02 (work in 334 progress), July 2017. 336 [I-D.ietf-netconf-yang-push] 337 Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen- 338 Nygaard, E., Bierman, A., and B. Lengyel, "YANG Datastore 339 Subscription", draft-ietf-netconf-yang-push-11 (work in 340 progress), October 2017. 342 Appendix A. Change Log 344 (To be removed by RFC editor prior to publication) 346 v01 348 o Minor revision on Subscription Decomposition 350 o Revised terminologies 352 o Removed most implementation related text 354 o Place holder of two sections: Subscription Management, and 355 Notifications on Subscription State Changes 357 Appendix B. Subscription Decomposition 359 Since Agents are invisible to the Collector, the Collector can only 360 subscribe to the Master. This requires the Master to: 362 1. expose the Global Capability that can be served by multiple 363 stream originators; 365 2. disassemble the Global Subscription to multiple Component 366 Subscriptions, and distribute them to the corresponding telemetry 367 sources; 369 3. notify on changes between portions of a subscription moving 370 between different Agents over time. 372 To achieve the above requirements, the Master need a Global 373 Capability description which is typically the YANG [RFC7950] data 374 model. This global YANG model is provided as the contract between 375 the Master and the Collector. Each Agent associating with the Master 376 owns a local YANG model to describe the Component Capabilities which 377 it can serve as part of the Global Capability. All the Agents need 378 to know the namespace associated with the Master. 380 The Master also need a data structure, typically a Resource-Location 381 Table, to keep track of the mapping between the resource and the 382 corresponding location of the Subscription Server which commits to 383 serve the data.When a Global Subscription request arrives, the Master 384 will firstly extract the filter information from the request. 385 Consequently, according to the Resource-Location Table, the Global 386 Subscription can be disassembled into multiple Component 387 Subscriptions, and the corresponding location can be associated. 389 The decision whether to decompose a Global Subscription into multiple 390 Component Subscriptions rests with the Resource-Location Table. A 391 Master can decide to not decompose a Global Subscription at all and 392 push a single stream to the receiver, because the location 393 information indicates the Global Subscription can be served locally 394 by the Master. Similarly, it can decide to entirely decompose a 395 Global Subscription into multiple Component Subscriptions that each 396 push their own streams, but not from the Master. It can also decide 397 to decompose the Global Subscription into several Component 398 Subscriptions and retain some aspects of the Global Subscription 399 itself, also pushing its own stream. 401 Component Subscriptions belong to the same Global Subscription MUST 402 NOT overlap. The combination of all Component Subscriptions MUST 403 cover the same range of nodes as the Global Subscription. Also, the 404 same subscription settings apply to each Component Subscription, 405 i.e., the same receivers, the same time periods, the same encodings 406 are applied to each Component Subscription per the settings of the 407 Global Subscription. 409 Each Component Subscription in effect constitutes a full-fledged 410 subscription, with the following constraints: 412 o Component subscriptions are system-controlled, i.e. managed by the 413 Master Node, not by the subscriber. 415 o Component subscription settings such as time periods, dampening 416 periods, encodings, receivers adopt the settings of their Global 417 Subscription. 419 o The life-cycle of the Component Subscription is tied to the life- 420 cycle of the Global Subscription. Specifically, terminating/ 421 removing the Global Subscription results in termination/removal of 422 Component Subscriptions. 424 o The Component Subscriptions share the same Subscription ID as the 425 Global Subscription. 427 Appendix C. Publication Composition 429 The Publisher collects data and encapsulates the packets per the 430 component subscription. There are several potential encodings, 431 including XML, JSON, CBOR and GPB. The encoding of the data records 432 follows the YANG schema, so that the composition at the Receiver can 433 benefit from the structured and hierarchical data instance. The 434 Collector may be able to assemble many pieces of data associated with 435 one subscription, and can also deduce the missing pieces of data. 437 The Collector recognizes data records associated with one 438 subscription according the Subscription ID. Data records generated 439 per one subscription are assigned with the same Subscription ID. 441 For the time series data stream, records are produced periodically 442 from each stream originator. The message arrival time varies because 443 of the distributed nature of the publication. The receiver assembles 444 data generated at the same time period based on the recording time 445 consisted in each data record. In this case, time synchronization is 446 required for all the steam originators. 448 Appendix D. Subscription Management 450 A Global Subscription can be rejected for multiple reasons. Some are 451 related to the Subscription Decomposition and Component Subscription. 452 New error codes are defined to indicate why a datastore subscription 453 attempt has failed. The subscription result with the failure reason 454 is returned as part of the RPC response. 456 Appendix E. Notifications on Subscription State Changes 458 Each component subscription maintains its own subscription state and 459 is responsible for sending its own OAM notifications (for example, 460 when the component subscription is suspended or when it can resume). 462 TBD. 464 Appendix F. Configured Subscription and Call Home 466 TBD. Only about the message layer which is transport independent. 468 Authors' Addresses 470 Tianran Zhou 471 Huawei 472 156 Beiqing Rd., Haidian District 473 Beijing 474 China 476 Email: zhoutianran@huawei.com 478 Guangying Zheng 479 Huawei 480 101 Yu-Hua-Tai Software Road 481 Nanjing, Jiangsu 482 China 484 Email: zhengguangying@huawei.com 486 Eric Voit 487 Cisco Systems 488 United States of America 490 Email: evoit@cisco.com 492 Alexander Clemm 493 Huawei 494 2330 Central Expressway 495 Santa Clara, California 496 United States of America 498 Email: alexander.clemm@huawei.com 499 Andy Bierman 500 YumaWorks 501 United States of America 503 Email: andy@yumaworks.com