YumaWorks

andy@yumaworks.com

This document describes NETCONF protocol enhancements to improve data retrieval capabilities.

There is a need for standard mechanisms to allow NETCONF application designers to retrieve data from NETCONF servers more efficiently.

This document attempts to address the following problems with NETCONF data retrieval mechanisms.

The NETCONF <get> operation allows a client to retrieve data from the server but it returns all data, including configuration datastore nodes. The <get‑config> operation already returns all configuration datastore nodes. It was originally thought that <get> should return all nodes so the client would not have to correlate configuration and non-configuration data nodes, since they would be mixed together in the reply. Operational experience has shown that the <get> operation without reasonable filters to reduce the returned data can significantly degrade device performance and return enormous XML instance documents in the <rpc‑reply>.

The NETCONF protocol has no standard mechanisms to indicate to a client when a datastore was last modified, or to allow a client to retrieve data only if it has been modified since a specified time. This makes polling applications very inefficient because they will regularly burden the server and the network and themselves with retrieval and processing requests for data that has not changed.

Sometimes the client application wants to discover what data exists on the server, particularly list entries. There is a need for a simple mechanism to retrieve just the key leaf nodes within a subtree. The NETCONF subtree filtering mechanism does provide a very complex way for the client to request just key leafs for specific list entries. A simpler mechanism is needed which will allow the client to discover the list instances present.

NETCONF filters allow the client to select specific sub-trees within the conceptual datastore on the server. However, sometimes the client does not really need the entire subtree, which may contain many nested list entries, and be very large. There is sometimes a need to limit the depth of the sub-trees retrieved from the server. A consistent and simple algorithm for determining what data nodes start a new level is needed.

The NETCONF <get> and <get‑config> operations use a hard-coded content filtering mechanism. They use a "type" XML attribute to indicate which of two filter specification types they support, and a "select" XML attribute if the :xpath capability is supported and an XPath expression filter specification is provided. This design does not allow additional content filter specification types to be supported by an implementation. It does not allow the standard to be easily extended in a modular fashion. In addition, this design does not allow YANG statements to be used to properly describe the protocol operation. The special "get‑filter‑element‑attributes" YANG extension in the ietf-netconf module is not extensible, and it does not really count as proper YANG, since this extension is outside the YANG language definition.

The operational data nodes returned by the server can sometimes represent server state parameters which may be derived from different sources. For example, an operational node representing the current date and time in use on a system might be derived from the Network Time Protocol (NTP) or from an action operation to set the current time. A list representing a routing entries in use in a router might include entries learned from a routing protocol and entries statically configured in the running datastore. There is a need for standard mechanisms to: identify data-model specific sources of operational data. identify which nodes in a datastore that the server should maintain data source information allow the client to retrieve the data source information.

This document defines a new NETCONF protocol operation called <get2> to address the deficiencies described in the previous section. It can be implemented existing NETCONF servers without requiring a change in the protocol version.

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, .

The following terms are defined in : candidate configuration datastore client configuration data datastore configuration datastore protocol operation running configuration datastore server startup configuration datastore

The following terms are defined in : anyxml container data node key leaf leaf leaf-list list presence container (or P-container) non-presence container (or NP-container)

The following terms are defined: non-configuration data node: a data node which is not a configuration data node, i.e. config=false. operational datastore: the collection of all conceptual YANG data nodes which represent non-configuration data. This conceptual datastore also includes ancestor container and list nodes for any nested non-configuration data nodes, as well as list keys for any list data nodes in this datastore. operational data node: A data node which is is contained within the operational datastore. Ancestor container, list, and key leaf nodes for any nested non-configuration nodes are only operational data nodes if they are also non-configuration data nodes. depth filter: A mechanism implemented within the NETCONF server to allow a client to retrieve only a limited number of levels within the a subtree, instead of retrieving the entire subtree. time filter: A mechanism implemented within the NETCONF server to allow a client to retrieve only data that has been modified since a specified data and time.

The <get2> operation is defined with a YANG "rpc" statement. A specific datastore is selected for the source of the retrieval operation. Several different types of filters are provided. Filters are combined in a conceptual "logical‑AND" operation, and are optional to use by the client. Not all filtering mechanisms are mandatory-to-implement for the server. The <get2> protocol operation contains the following input parameters: source: A container indicating the conceptual datastore for the retrieval request. filter-spec: A choice indicating the content filter specification for the retrieval request. keys-only: A leaf indicating that only the key leafs, combined with other filtering criteria, should be returned. if-modified-since: A leaf indicating the time filter specification for the retrieval request, according to the procedures in . depth: A leaf indicating the subtree depth level for the retrieval request, according to the procedures in . with-defaults: A leaf indicating the type of defaults handling requested, according to procedures in . with-timestamps: A leaf indicating that "last‑modified" XML attributes are requested, encoded according to the schema in . with-data-sources: A leaf indicating that "data‑source" XML attributes are requested, encoded according to the schema in .

A depth filter indicates how many subtree levels should be returned in the <rpc‑reply>. This filter is specified with the "depth" input parameter for the <get2> protocol operation. The default "0" indicates that all levels from the requested subtrees should be returned. A new level is started for each YANG data node within the requested subtree. All top level data nodes are considered to be child nodes (level 1) of a conceptual <config> root. If no content filters are provided, then level 1 is considered to include all top-level data nodes within the source datastore. Otherwise only the levels in selected subtrees will be considered, and not any additional top-level data nodes. If the depth requested is equal to "1", then only the requested data nodes (or top-level data nodes) will be returned. This mechanism can be used to detect the existence of containers and list entries within a particular subtree, without returning any of the descendant nodes. Higher depth values indicates the number of descendant nodes to include in the response. For example, if the depth requested is equal to "2", then only the requested data nodes (or top-level data nodes) and their immediate child data nodes will be returned.

A time filter indicates that only data which has been modified since the indicated date and time should be included in the reply. If this feature is supported, then the server will maintain a last-modified timestamp for the source datastore. It MAY support additional nested timestamps for data nodes within the datastore. When a request containing the "if‑modified‑since" parameter is received, the server will compare that timestamp to the last-modified timestamp for the source datastore. If it is greater than the specified value then data may be returned (depending on other filters). If the datastore timestamp value is less than or equal to the specified value, then an empty <data> element will be returned in the <rpc‑reply>. If the server maintains "last‑modified" timestamps for any data nodes within the source datastore then the same type of comparison will be done for the data node to determine if it should be included in the response. If no "last‑modified" timestamp is maintained for a data node, then the server will use the "last‑modified" timestamp for its nearest ancestor, or for the datastore itself if there are none.

Operational data source reporting is supported if the server advertises the "data‑source" feature. If the "with‑data‑sources" parameter is present in the <get2> request, and the server supports the "data‑source" feature, then data source reporting will be done for the applicable nodes. An operational data source applies only to operational data nodes, and only if the "data‑source" YANG extension statement defined in is present in the YANG data definition statement for the data node. If the "data‑source" extension applies to a data node, then a server that implements the "data‑source" feature is expected to return the "data‑source" XML attribute for that node.

Operational data sources are defined with YANG identity statements. The YANG module in contains the base identity "data‑sources", and a few common data sources: server: normal case: the server instrumentation is the source of the operational data value. running: the operational data value is derived from a value in the running configuration datastore. operation: the operational data value is derived from a direct or side effect of a client-initiated protocol operation. ntp: the operational data value is derived from NTP information dns: the operational data value is derived from DNS information Other modules can define new data source identities, such as the "thfp" protocol in the "example‑get2" module. The "data‑source" YANG extension is defined in . It is used within other YANG modules to identify which operational data nodes should have data source information maintained by the server.

The following XML Schema document defines the "last‑modified" attribute, described within this document. This XSD is only relevant if the server supports the "timestamps" YANG feature within the "ietf‑netconf‑get2" YANG module. The "last‑modified" attribute uses the XSD data type "dateTime", in accordance with Section 3.2.7.1 of XML Schema Part 2: Datatypes. This is equivalent to the YANG data type "date‑and‑time". <CODE BEGINS> file "last‑modified.xsd"

This schema defines the syntax for the "last-modified" attribute described within this document. This attribute indicates the date and time when a modification was last detected by the server for the datastore or data node corresponding to the XML element containing this attribute. ]]>

<CODE ENDS>

The following XML Schema document defines the "data‑source" attribute, described within this document. This attribute uses the XSD data type "QName", in accordance with Section 3.2.18.1 of XML Schema Part 2: Datatypes. This is an XML encoding of the YANG "identityref" data type: the module namespace statement value for the YANG module containing the identity statement is represented in the"namespace‑name" part. the identity name is represented in the local part. <CODE BEGINS> file "data‑source.xsd"

This schema defines the syntax for the "data-source" attribute described within this document. This attribute indicates the identity statement for the configuration source for the data node corresponding to the XML element containing this attribute. ]]>

<CODE ENDS>

RFC Ed.: update the date below with the date of RFC publication and remove this note. <CODE BEGINS> file "ietf-netconf-data-source@2012-10-09.yang"

WG List: WG Chair: Mehmet Ersue WG Chair: Bert Wijnen Editor: Andy Bierman "; description "This module contains a collection of YANG definitions for identifying the data source of operational data nodes. Copyright (c) 2012 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices."; // RFC Ed.: replace XXXX with actual RFC number and remove this // note. // RFC Ed.: remove this note // Note: extracted from draft-bierman-netconf-get2-02.txt // RFC Ed.: update the date below with the date of RFC publication // and remove this note. revision "2012-10-09" { description "Initial revision."; reference "RFC XXXX: The NETCONF Operation"; } /* Extensions */ extension data-source { description "This extension indicates that the data node containing it is expected to support the 'data-source' XML attribute in replies for the operation. This extension is ignored if it is not within a YANG data definition statement for an operational data node."; } /* Identities */ identity data-sources { description "Base identity for all data sources."; } identity server { base data-sources; description "Indicates the server instrumentation as the data source."; } identity running { base data-sources; description "Indicates the running datastore as the data source."; } identity operation { base data-sources; description "Indicates a client-initiated protocol operation as the data source."; } identity ntp { base data-sources; description "Indicates the NTP protocol as the data source."; reference "RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification"; } identity dns { base data-sources; description "Indicates the DNS protocol as the data source."; // reference TBD } } ]]>

<CODE ENDS>

This module imports the "with‑defaults‑parameters" grouping from . Several YANG features are imported from . Some data types are imported from . RFC Ed.: update the date below with the date of RFC publication and remove this note. <CODE BEGINS> file "ietf-netconf-get2@2012-10-09.yang"

WG List: WG Chair: Mehmet Ersue WG Chair: Bert Wijnen Editor: Andy Bierman "; description "This module contains a collection of YANG definitions for the retrieval of information from a NETCONF server. Copyright (c) 2012 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices."; // RFC Ed.: replace XXXX with actual RFC number and remove this // note. // RFC Ed.: remove this note // Note: extracted from draft-bierman-netconf-get2-02.txt // RFC Ed.: update the date below with the date of RFC publication // and remove this note. revision "2012-10-09" { description "Initial revision."; reference "RFC XXXX: The NETCONF Operation"; } /* Features */ feature timestamps { description "This feature indicates that the server implements the operations parameters which require last modification timestamps to be maintained by the server. If this feature is advertised then one global 'last-modified' timestamp for the entire running datastore MUST be supported. The server MAY support additional timestamps for additional datastores and data nodes within a datastore. The 'with-timestamps' parameter can be used to identify which data nodes support a last-modified-time timestamp."; } feature with-defaults { description "This feature indicates that the server supports the 'with-defaults' parameter for the operation. A NETCONF server SHOULD support this feature."; reference "RFC 6243: With-defaults Capability for NETCONF"; } feature data-sources { description "This feature indicates that the server supports the 'with-data-sources' parameter for the operation. A NETCONF server SHOULD support this feature."; reference "RFC XXXX: The NETCONF Operation"; } /* Protocol Operations */ rpc get2 { description "Retrieve NETCONF datastore information"; input { container source { choice datastore-source { default running; description "The configuration source for the retrieval operation. The running configuration is the default choice if this parameter is not present."; leaf candidate { if-feature nc:candidate; type empty; description "The candidate configuration datastore is the retrieval source."; } leaf running { type empty; description "The running configuration datastore is the retrieval source."; } leaf operational { type empty; description "The collection of non-configuration data nodes supported by the server is the retrieval source."; } leaf startup { if-feature nc:startup; type empty; description "The startup configuration datastore is the retrieval source."; } leaf url { if-feature nc:url; type inet:uri; description "The URL-based configuration is the retrieval source."; } } } choice filter-spec { anyxml subtree-filter { description "This parameter identifies the portions of the target datastore to retrieve."; reference "RFC 6241, Section 6."; } leaf xpath-filter { if-feature nc:xpath; type yang:xpath1.0; description "This parameter contains an XPath expression identifying the portions of the target datastore to retrieve."; } } leaf keys-only { type empty; description "This parameter selects only data nodes which are key leaf nodes. Parent container and list nodes are also returned, but no other leafs, or any leaf-lists will be included in the reply."; } leaf if-modified-since { if-feature timestamps; type yang:date-and-time; description "This parameter selects only data nodes which have been modified since the specified time."; } leaf depth { type uint32; default 0; description "This parameter selects how many conceptual sub-tree levels should be returned in the . If this parameter is equal to '0', then entire subtrees will be returned. If this parameter is greater than '0', then only the specified number of subtree levels will be returned."; reference "RFC XXXX, section 2.1."; } uses ncwd:with-defaults-parameters { if-feature with-defaults; description "This parameter controls the retrieval of default values."; reference "RFC 6243: With-defaults Capability for NETCONF"; } leaf with-timestamps { if-feature timestamps; type empty; description "This parameter will cause the server to return XML attributes identifying the last modification time within one or more elements within the ."; reference "RFC XXXX, sections 2.2 and 3."; } leaf with-data-sources { if-feature data-sources; when "../source/operational"; type empty; description "This parameter will cause the server to return XML attributes identifying the operational data sources for elements within the that support the 'data-source' YANG extension statement."; reference "RFC XXXX, sections 2.2 and 3."; } } output { anyxml data { description "Copy of the requested datastore subset which matched the filter criteria (if any). An empty data container indicates that the request did not produce any results."; } } } } ]]>

<CODE ENDS>

This document registers a URI in the IETF XML registry . Following the format in RFC 3688, the following 2 registrations are requested to be made.

This document registers 2 YANG modules in the YANG Module Names registry .

This document does not introduce any new security concerns in addition to those specified in , section 9.

removed subtree-filter YANG feature changed depth filter to exactly match the XML layering renamed filter to subtree-filter renamed select to xpath-filter added some new examples

added operational data source support added 'ietf‑netconf‑data‑source' module clarified terminology

Harvard University In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. YANG is a data modeling language used to model configuration and state data manipulated by the Network Configuration Protocol (NETCONF), NETCONF remote procedure calls, and NETCONF notifications. [STANDARDS TRACK] This document describes an IANA maintained registry for IETF standards which use Extensible Markup Language (XML) related items such as Namespaces, Document Type Declarations (DTDs), Schemas, and Resource Description Framework (RDF) Schemas. YumaWorks Tail-f Systems

The follow instances are assumed in the following examples.

The forests and trees are configured, which represent trees the company has planted and growing over time. The operational data (tree height) represents the data that the company monitors for each tree over time. There are 2 operational data sources for tree height data in this example: thfp: Tree Height Finder Protocol manual: Manual measurement by work crews The north forest was measured with the mythical Tree Height Finder protocol and the south forest was measured manually.

This example simply retrieves the "forests" subtree data from the operational datastore.

north 3 birch 41.013 ash 16.523 maple 51.204 south 2 banyan 91.433 palm 83.439 ]]>

In this example, the running datastore was last modified at "2012‑09‑09T01:43:27Z" because the forest named "north" was modified at this time. The forest named "north" was last modified after the specified "if‑modified‑since" timestamp. The forest named "south" was last modified before the specified "if‑modified‑since" timestamp. The server maintains a last-modified timestamp for the running datastore and the "forest" list entries. The client is also requesting that timestamps be returned for the nodes that have been modified. If any part of the "forest" subtree is modified then this timestamp will be updated.

2012-09-09T01:43:27Z north birch hillside ash southwest pasture maple east meadow ]]>

In this example the client has changed the if-modified-since timestamp to a time in the future. No "forest" list entry has been modified since this time so an empty data node is returned. Note that the last-modified timestamp is returned for the node representing the datastore, even though no data nodes have been modified since the specified time. This allows the client to easily retrieve the last-modified timestamp for the entire datastore.

2012-09-09T03:43:27Z ]]>

This example retrieves the names-only from the "forests" subtree in the running datastore. The default source (running) is used. The default depth="0" is used to retrieve all subtree levels. The xpath-filter is used instead of the subtree-filter Whitespace added to xpath-filter element for display purposes only

/ex:forests north birch ash maple south banyan palm ]]>

This example retrieves the "trees" node to determine which forests have any trees. Only 1 subtree level is requested, instead of the default of all levels. The default source (running) is used. The depth parameter is set to "1" to only retrieve the requested layer (trees) and its ancestor nodes and the configuration leaf nodes from each "forest" entry.

1 north south ]]>

This example retrieves the names-only from the "forest" list within the "forests" subtree, in the running datastore. Only 3 subtree levels are requested, instead of the default of all levels. The default source (running) is used. The depth parameter is set to "3" to only retrieve the requested layer (forests), its child nodes (forest), and the key leaf nodes from each "forest" entry. Without the "keys‑only" parameter, other leafs from the "forest" list would be returned as well.

3 north south ]]>

This example simply retrieves the "forests" subtree data from the operational datastore, but requesting that data-source XML attributes be added as required in the reply.

north 3 birch 41.013 ash 16.523 maple 51.204 south 2 banyan 91.433 palm 83.439 ]]>

This example shows how the data-source reporting can be used with a real YANG module. The ietf-system module defined in contains an operational data node called "current‑datetime". The data source for this node can either be NTP or the "set‑current‑datetime" operation defined in the module. To implement data source reporting, the "data‑source" extension needs to added to the "current‑datetime" leaf as follows:

The following example shows the retrieval of the "current‑datetime" leaf if the data source is NTP. The extra whitespace shown for the "current‑datetime" leaf is for display purposes only.

2012-09-09T01:11:27Z ]]>

The following example shows the retrieval of the "current‑datetime" leaf if the data source is the "set‑current‑datetime" operation. The extra whitespace shown for the "current‑datetime" leaf is for display purposes only.

2012-09-09T01:11:27Z ]]>