idnits 2.17.1 draft-zheng-netconf-udp-pub-channel-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (July 2, 2017) is 2482 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '1' on line 366 -- Looks like a reference, but probably isn't: '2' on line 366 -- Looks like a reference, but probably isn't: '3' on line 364 -- Looks like a reference, but probably isn't: '6' on line 364 -- Looks like a reference, but probably isn't: '7' on line 366 -- Looks like a reference, but probably isn't: '4' on line 366 -- Looks like a reference, but probably isn't: '5' on line 366 -- Looks like a reference, but probably isn't: '8' on line 366 -- Looks like a reference, but probably isn't: '9' on line 362 -- Looks like a reference, but probably isn't: '0' on line 366 ** Obsolete normative reference: RFC 7049 (Obsoleted by RFC 8949) == Outdated reference: A later version (-26) exists of draft-ietf-netconf-subscribed-notifications-02 == Outdated reference: A later version (-25) exists of draft-ietf-netconf-yang-push-07 == Outdated reference: A later version (-01) exists of draft-voit-netconf-notification-messages-00 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 11 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NETCONF G. Zheng 3 Internet-Draft T. Zhou 4 Intended status: Standards Track A. Clemm 5 Expires: January 3, 2018 Huawei 6 July 2, 2017 8 UDP based Publication Channel for Streaming Telemetry 9 draft-zheng-netconf-udp-pub-channel-00 11 Abstract 13 This document describes a UDP-based publication channel for streaming 14 telemetry use to collect data from devices. A new shim header is 15 proposed to facilitate the distributed data collection mechanism 16 which directly pushes data from line cards to the collector. Because 17 of the lightweight UDP encapsulation, higher frequency and better 18 transit performance can be achieved. 20 Requirements Language 22 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 23 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 24 document are to be interpreted as described in RFC 2119 [RFC2119]. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on January 3, 2018. 43 Copyright Notice 45 Copyright (c) 2017 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 61 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 4 63 4. UDP Transport for Publication Channel . . . . . . . . . . . . 6 64 4.1. Data Format . . . . . . . . . . . . . . . . . . . . . . . 6 65 4.2. Options . . . . . . . . . . . . . . . . . . . . . . . . . 8 66 4.2.1. Reliability Option . . . . . . . . . . . . . . . . . 8 67 4.2.2. Authentication Option . . . . . . . . . . . . . . . . 9 68 4.2.3. Encryption Option . . . . . . . . . . . . . . . . . . 9 69 4.3. Data encoding . . . . . . . . . . . . . . . . . . . . . . 9 70 5. YANG Data Model for Subscription Management . . . . . . . . . 9 71 6. Retransmission Request . . . . . . . . . . . . . . . . . . . 10 72 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 73 8. Security Considerations . . . . . . . . . . . . . . . . . . . 10 74 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 75 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 76 10.1. Normative References . . . . . . . . . . . . . . . . . . 10 77 10.2. Informative References . . . . . . . . . . . . . . . . . 11 78 Appendix A. An Appendix . . . . . . . . . . . . . . . . . . . . 11 79 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 81 1. Introduction 83 Streaming telemetry refers to sending a continuous stream of 84 operational data from a device to a remote receiver. This provides 85 an ability to monitor a network from remote and to provide network 86 analytics. Devices generate telemetry data and push that data to a 87 collector for further analysis. By streaming the data, much better 88 performance, finer-grained sampling, monitoring accuracy, and 89 bandwidth utilization can be achieved than with polling-based 90 alternatives. 92 Sub-Notif [I-D.ietf-netconf-subscribed-notifications] and YANG-Push 93 [I-D.ietf-netconf-yang-push] defines a mechanism that allows a 94 collector to subscribe to updates of YANG-defined data that is 95 maintained in a YANG [RFC7950] datastore. The mechanism separates 96 the management and control of subscriptions from the transport that 97 is used to actually stream and deliver the data. Two transports have 98 been defined so far, Netconf and Restconf/HTTP2. 100 While powerful in its features and general in its architecture, in 101 its current form the mechanism needs to be extended to stream 102 telemetry data at high velocity from devices that feature a 103 distributed architecture. Specifically, there are two aspects that 104 need to be addressed: 106 1. The transports that have been defined so far, Netconf and HTTP2, 107 are ultimately based on TCP and lack the efficiency needed to 108 stream data continuously at high velocity. A lighterweight, more 109 efficient transport, e.g. a transport based on UDP is needed. 111 * Firstly, data collector will suffer a lot of TCP connections 112 from many line cards equipped on different devices. 114 * Secondly, as no connection state needs to be maintained, UDP 115 encapsulation can be easily implemented by hardware which will 116 further improve the performance. 118 * Thirdly, because of the lightweight UDP encapsulation, higher 119 frequency and better transit performance can be achieved, 120 which is important for streaming telemetry. 122 2. The current design involves a single push server. In the case of 123 data originating from multiple line cards, the design requires 124 data to be internally forwarded from those line cards to the push 125 server, presumably on a main board, which then combines the 126 individual data items into a single consolidated stream. This 127 centralized data collection mechanism can result in a performance 128 bottleneck, especially when large amounts of data are involved. 129 What is needed instead is support for a distributed mechanism 130 that allows to directly push multiple individual substreams, e.g. 131 one from each line card, without needing to first pass them 132 through an additional processing stage for internal 133 consolidation, but still allowing those substreams to be managed 134 and controlled via a single subscription. 136 This document specifies a distributed data collection mechanism which 137 can directly push data from line cards to a collector by using a UDP 138 based publication channel. Specifically, the following are 139 specified: 141 o A higher-performance transport option for YANG-Push that leverages 142 UDP. 144 o Extensions to YANG-Push's subscription model that allow a single 145 subscription to control multiple internal data originators that 146 each generate their own independent telemetry streams. Note: 147 Because the ability to support multiple streams via a single 148 subscription might be applicable to other transports as well, this 149 aspect might be split into a separate specification in future 150 revisions of this draft. 152 While this document will focus on the data publication channel, the 153 subscription can be used in conjunction with the mechanism proposed 154 in [I-D.ietf-netconf-yang-push] with necessary extensions. 156 2. Terminology 158 Streaming telemetry: refers to sending a continuous stream of 159 operational data from a device to a remote receiver. This provides 160 an ability to monitor a network from remote and to provide network 161 analytics. 163 Component subscription: A subscription that defines the data from 164 each individual entity which is managed and controlled by a single 165 subscription server. 167 Subscription agent: An agent that streams telemetry data per the 168 terms of a component subscription. 170 3. Solution Overview 172 The typical distributed data collection solution is shown in figure 173 1. The subscription server located in the main board receives the 174 subscription requests or configurations. It may be colocated, not 175 necessary, with a Netconf server which interacts with outside 176 clients. When receiving a subscription request, the subscription 177 server decomposes the subscription into multiple component 178 subscriptions, each involving data from a separate internal telemetry 179 source, for example a line card. The component subscriptions are 180 distributed within the device to the subscription agents located in 181 line cards. Subsequently, each line card generates its own stream of 182 telemetry data, collecting and encapsulating the packets per the 183 component subscription and streaming it to the designated data 184 collector. 186 The publication channel supports the reliable data streaming, for 187 example for some alarm events. The subscriber has the option of 188 deducing the packet loss and the disorder based on the information 189 carried by the notification data. And the subscriber will decide the 190 behavior to request retransmission. The subscriber can send the 191 retransmission request to the subscriber server for further 192 processing. 194 Subscription server and subscription agents interact with each other 195 in several ways: 197 o Subscription agents need to have a registration or announcement 198 handshake with the subscription server, so the subscription server 199 is aware of them and of lifecycle events (such as subscription 200 agents appearing and disappearing). 202 o The subscription server relays the component subscriptions to the 203 subscription agents. 205 o The subscription agents indicate status of component subscriptions 206 to the subscription server. The status of the overall "master" 207 subscription is maintained by the subscription server. The 208 subscription server is also responsible for notifying the 209 subscriber in case of any problems of component subscriptions. 211 The rest of the draft describes the UDP based publication channel. 213 retransmission + + 214 request | | subscription 215 +------------------------+ 216 | | | Main Board| 217 | +--v----v--------+ | 218 | | subscription | | 219 | | server | | 220 | +--+----+-----+--+ | 221 | | | | | internal 222 +------------------------+ subscription 223 | | | distribution 224 +---------------+ | +--------------+ 225 | | | 226 +------------------+ +------------------+ +------------------+ 227 | | | | | | | | | 228 | +-------v------+ | | +------v-------+ | | +-----v--------+ | 229 | | subscription | | | | subscription | | | | subscription | | 230 | | agent | | | | agent | | | | agent | | 231 | +--------------+ | | +--------------+ | | +--------------+ | 232 | Line Card 1 | | Line Card 2 | | Line Card n | 233 +---------+--------+ +--------+---------+ +----------+-------+ 234 | | | 235 | | Publication Channel | 236 +--------------+ | +-----------------+ 237 | | | 238 +-v-----v-----v-+ 239 | | 240 | Collector | 241 | | 242 +---------------+ 244 4. UDP Transport for Publication Channel 246 In [I-D.voit-netconf-notification-messages], the transport 247 independent message header is proposed for the notification use. The 248 following shim header refers to and implements that message header 249 definition. 251 4.1. Data Format 253 The data format of the UDP based based publication transport is shown 254 as follows. 256 0 1 2 3 257 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 258 +---------------------------------------------------------------+ 259 | UDP Header ~ 260 +-------+---------------+-----------------------+---------------+ 261 | Vers. | Flag | Reserved | Msg-Gen-ID | 262 +-------+---------------+-----------------------+---------------+ 263 | Device ID | 264 +---------------------------------------------------------------+ 265 | Timestamp (Seconds) | 266 +---------------------------------------------------------------+ 267 | Timestamp (MicroSeconds) | 268 +---------------------------------------------------------------+ 269 | Options ~ 270 +---------------------------------------------------------------+ 271 | Message Content ~ 272 +---------------------------------------------------------------+ 274 Right after the UDP header, a simple inform header is attached to 275 carry the necessary information with regard to the streaming mode. 277 o The Vers. field represents the PDU encoding version. The initial 278 version value is 0. 280 o The Flag is a bitmap indicating what features this packet has and 281 the corresponding options attached. Each bit associates to one 282 feature and one option. When the bit is set to 1, the associated 283 feature is enabled and the option is attached. The sequence of 284 the presence of the options also follows the position in the 285 bitmap. Right now 3 options are specified. 287 * bit 0, the reliability option; 289 * bit 1, the authentication option; 291 * bit 2, the encryption option; 293 * other bits are reserved. 295 o The Msg-Gen-ID stands for the message generator ID. It identifies 296 the process, either on main board or line cards, which created the 297 packet. 299 o The Device ID identifies the device with an global unique number 300 that will not repeat among all the managed devices. It can be 301 generated by some unique device information like MAC address. 303 o The Timestamp, including the second part and the microsecond part, 304 indicate the time the message was packaged and sent to the 305 receiver. The Timestamp is defined per RFC 3339. 307 o The details of the Options will be described in the respective 308 sections. 310 After the inform header is the real content which is encoded. The 311 actual encoding is based on the subscription, e.g., in binary with 312 GPB or CBOR [RFC7049]. 314 More details of the content encoding is TBD. 316 4.2. Options 318 4.2.1. Reliability Option 320 The UDP based publication transport described in this document 321 provides two streaming modes, the reliable mode an the unreliable 322 mode, for different SLA (Service Level Agreement) and telemetry 323 requirements. 325 In the unreliable streaming mode, the line card pushes the 326 encapsulated data to the data collector without any sequence 327 information. So the subscriber does not know whether the data is 328 correctly received or not. Hence no retransmission happens. 330 The reliable streaming mode provides sequence information in the UDP 331 packet, based on which the subscriber can deduce the packet loss and 332 disorder. Then the subscriber can decide whether to request the 333 retransmission of the lost packets. 335 In most case, the unreliable streaming mode is preferred. Because 336 the reliable streaming mode will cost more network bandwidth and 337 precious device resource. Different from the unreliable streaming 338 mode, the line card cannot remove the sent reliable notifications 339 immediately, but to keep them in the memory for a while. Reliable 340 notifications may be pushed multiple times, which will increase the 341 traffic. When choosing the reliable streaming mode or the unreliable 342 streaming mode, the operate need to consider the reliable requirement 343 together with the resource usage. 345 When the reliability flag is set to 1. The following option will be 346 attached 347 0 1 2 3 348 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 349 +---------------------------------------------------------------+ 350 | Notification ID | 351 +---------------------------------------------------------------+ 352 | Previous Notification ID | 353 +---------------------------------------------------------------+ 355 The notification ID is generated continuously by the message 356 generator. Different subsrcibers share the same notification ID 357 sequence. Current ID and previous ID will be added in the packets. 358 For example, there are two subscriber A and B, 360 o Notification IDs for the generator are : 1,2,3,[4,5],6,7,8,9, in 361 which Subscriber A subscribes [1,2,3,6,7], Subscriber B subscribes 362 [1,2,4,5,7,8,9]. 364 o A will receive : [0,1][1,2][2,3][3,6][6,7] 366 o B will receive : [0,1][1,2][2,4][4,5][5,7][7,8] 368 4.2.2. Authentication Option 370 TBD 372 4.2.3. Encryption Option 374 TBD 376 4.3. Data encoding 378 Subscribed data can be encoded in GPB, CBOR, XML or JSON format. It 379 is conceivable that additional encodings may be supported as options 380 in the future. This can be accomplished by augmenting the 381 subscription data model with additional identity statements used to 382 refer to requested encodings. 384 5. YANG Data Model for Subscription Management 386 To enable the UDP based publication transport, the subscription 387 configuration need to provide necessary information. The 388 subscription management YANG Model extends ietf-subscribed- 389 notifications described [I-D.ietf-netconf-yang-push] 390 +--rw subscription-config {configured-subscriptions}? 391 | + ... 392 | +--rw receivers 393 | | + ... 394 | | +--rw protocol? transport-protocol 395 | | | +--rw udp-transport-type? udp-transport-type 396 | | | +--rw reliable? 397 | | | +--rw authentication? 398 | | | +--rw encryption? 400 As in the above YANG tree, when the transport protocol is set to UDP, 401 retries indicates the maximum retry times of the reliable streaming 402 mode, and the timeout indicates the time out for retry in reliable 403 streaming mode. 405 TBD. Note this YANG tree just to show we need to extend subcription 406 mode, including the configurations and the RPCs. More details will 407 be added later. 409 6. Retransmission Request 411 TBD 413 7. IANA Considerations 415 TBD 417 8. Security Considerations 419 9. Acknowledgements 421 The authors of this documents would like to thank Eric Voit, Tim 422 Jenkins, and Huiyang Yang for the initial comments. 424 10. References 426 10.1. Normative References 428 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 429 Requirement Levels", BCP 14, RFC 2119, 430 DOI 10.17487/RFC2119, March 1997, 431 . 433 [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object 434 Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, 435 October 2013, . 437 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 438 RFC 7950, DOI 10.17487/RFC7950, August 2016, 439 . 441 10.2. Informative References 443 [I-D.ietf-netconf-subscribed-notifications] 444 Voit, E., Clemm, A., Prieto, A., Nilsen-Nygaard, E., and 445 A. Tripathy, "Custom Subscription to Event Notifications", 446 draft-ietf-netconf-subscribed-notifications-02 (work in 447 progress), April 2017. 449 [I-D.ietf-netconf-yang-push] 450 Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen- 451 Nygaard, E., Bierman, A., and B. Lengyel, "Subscribing to 452 YANG datastore push updates", draft-ietf-netconf-yang- 453 push-07 (work in progress), June 2017. 455 [I-D.voit-netconf-notification-messages] 456 Voit, E., Bierman, A., Clemm, A., and T. Jenkins, 457 "Notification Message Headers and Bundles", draft-voit- 458 netconf-notification-messages-00 (work in progress), April 459 2017. 461 Appendix A. An Appendix 463 Authors' Addresses 465 Guangying Zheng 466 Huawei 467 Nanjing, Jiangsu 468 China 470 Fax: + 471 Email: zhengguangying@huawei.com 473 Tianran Zhou 474 Huawei 475 156 Beiqing Rd., Haidian District 476 Beijing 477 China 479 Email: zhoutianran@huawei.com 480 Alexander Clemm 481 Huawei 482 2330 Central Expressway 483 Santa Clara, California 484 USA 486 Email: alexander.clemm@huawei.com