<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
]>

<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc toc="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc iprnotified="no"?>
<?rfc strict="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info"
     docName="draft-ietf-detnet-data-plane-framework-03"
     ipr="trust200902"
     submissionType="IETF">
  <front>
    <title abbrev="DetNet Data Plane Framework">
    DetNet Data Plane Framework</title>

    <author role="editor" fullname="Bal&aacute;zs Varga" initials="B." surname="Varga">
      <organization>Ericsson</organization>
      <address>
        <postal>
          <street>Magyar Tudosok krt. 11.</street>
          <city>Budapest</city>
          <country>Hungary</country>
          <code>1117</code>
        </postal>
        <email>balazs.a.varga@ericsson.com</email>
      </address>
    </author>

    <author fullname="J&aacute;nos Farkas" initials="J." surname="Farkas">
      <organization>Ericsson</organization>
      <address>
        <postal>
          <street>Magyar Tudosok krt. 11.</street>
          <city>Budapest</city>
          <country>Hungary</country>
          <code>1117</code>
        </postal>
        <email>janos.farkas@ericsson.com</email>
      </address>
    </author>

    <author fullname="Lou Berger" initials="L." surname="Berger">
      <organization>LabN Consulting, L.L.C.</organization>
      <address>
        <email>lberger@labn.net</email>
      </address>
    </author>
    <author fullname="Don Fedyk" initials="D." surname="Fedyk">
      <organization>LabN Consulting, L.L.C.</organization>
      <address>
        <email>dfedyk@labn.net</email>
      </address>
    </author>

    <author fullname="Andrew G. Malis" initials="A.G." surname="Malis">
      <organization>Independent</organization>
      <address>
        <email>agmalis@gmail.com</email>
      </address>
    </author>

    <author fullname="Stewart Bryant" initials="S." surname="Bryant">
      <organization>Futurewei Technologies</organization>
      <address>
        <email>stewart.bryant@gmail.com</email>
      </address>
    </author>

    <author fullname="Jouni Korhonen" initials="J." surname="Korhonen">
      <!--organization abbrev="Nordic">Nordic Semiconductor</organization-->
      <address>
        <email>jouni.nospam@gmail.com</email>
      </address>
    </author>
    <date />
    <workgroup>DetNet</workgroup>

    <abstract>
      <t>
        This document provides an overall framework for the DetNet
        data plane.  It covers concepts and considerations that
        are generally common to any Deterministic Networking data plane
        specification.
      </t>
    </abstract>
  </front>

  <middle>
    <section title="Introduction" anchor="sec_intro">
      <t>
	DetNet (Deterministic Networking) provides a capability to carry
	specified unicast or multicast data flows for real-time applications
	with extremely low packet loss rates and assured maximum end-to-end
	delivery latency.  A description of the general background and concepts
	of DetNet can be found in <xref target="I-D.ietf-detnet-architecture"/>.
      </t>
       <t>
        This document describes the concepts needed by any DetNet data plane specification
        and provides considerations for any corresponding implementation.
	It covers the building blocks that provide the DetNet service, the
	DetNet service sub-layer and the DetNet forwarding sub-layer functions
	as described in the DetNet Architecture.
      </t>
      <t>
	The DetNet Architecture models the DetNet related data plane functions
	decomposed into two sub-layers: a service sub-layer and a forwarding
	sub-layer.  The service sub-layer is used to provide DetNet service
	protection and reordering. The forwarding sub-layer leverages 
	Traffic Engineering mechanisms and provides congestion protection 
	(low loss, assured latency, and limited out-of-order delivery).
      </t>
      <t>
        As part of the service sub-layer functions, this document describes
        typical DetNet node data plane operation. It describes the function
        and operation of the Packet Replication (PRF) Packet Elimination
        (PEF) and the Packet Ordering (POF) functions within the service
        sub-layer. Furthermore, it also describes the forwarding sub-layer.
      </t>
      <t>
	DetNet flows may be carried over network technologies that can provide
	the DetNet required service characteristics.  For example, DetNet MPLS flows
	can be carried over IEEE 802.1 Time Sensitive Network (TSN) <xref
		target="IEEE802.1TSNTG"/> sub-networks. However, IEEE 802.1 TSN
	support is not required and some of the DetNet benefits can be gained
	by running over a data link layer that has not been specifically
	enhanced to support TSN.
      </t>
      <t>
	Different application flows (e.g., Ethernet, IP, etc.), can be mapped on top of
	DetNet.  DetNet can optionally reuse header information provided by, or
	shared with, applications. An example of shared header fields can be
	found in <xref target="I-D.ietf-detnet-ip"/>.
      </t>
      <t>
        This document also covers
        basic concepts related to the controller plane and Operations,
	Administration, and Maintenance (OAM). 
	Data plane OAM specifics are out of scope for this docuement. 
      </t>

    </section>

    <section title="Terminology">
      <section title="Terms Used in This Document">
        <t>
          This document uses the terminology established in the DetNet
          architecture <xref target="I-D.ietf-detnet-architecture"/>,
          and the reader is assumed to be familiar with that document
          and its terminology.
        </t>
      </section>

      <section title="Abbreviations">
        <t>
          The following abbreviations are used in this document:
          <list style="hanging" hangIndent="14">
            <t hangText="CW">Control Word.</t>
            <t hangText="d-CW">DetNet Control Word.</t>
            <t hangText="DetNet">Deterministic Networking.</t>
            <t hangText="DN">DetNet.</t>
            <t hangText="GRE">Generic Routing Encapsulation.</t>
            <t hangText="IPSec">IP Security.</t>
            <t hangText="L2">Layer 2.</t>
            <t hangText="LSR">Label Switching Router.</t>
            <t hangText="MPLS">Multiprotocol Label Switching.</t>
            <t hangText="MPLS-TE">Multiprotocol Label Switching - Traffic Engineering.</t>
            <t hangText="OAM">Operations, Administration, and Maintenance.</t>
            <t hangText="PEF">Packet Elimination Function.</t>
            <t hangText="PRF">Packet Replication Function.</t>
            <t hangText="PREOF">Packet Replication, Elimination and Ordering Functions.</t>
            <t hangText="POF">Packet Ordering Function.</t>
            <t hangText="PSN">Packet Switched Network.</t>
            <t hangText="PW">PseudoWire.</t>
            <t hangText="QoS">Quality of Service.</t>
            <t hangText="S-Label">DetNet "service" label.</t>			
            <t hangText="TDM">Time-Division Multiplexing.</t>
            <t hangText="TSN">Time-Sensitive Network.</t>
          </list>
        </t>
      </section>
    </section>  <!-- end of terminology -->

    <section title="DetNet Data Plane Overview" anchor="sec_dt_dp">
        <t>
        This document describes how application flows, or app-flows, are
        carried over DetNet networks. The DetNet Architecture, <xref
                target="I-D.ietf-detnet-architecture"/>, models the DetNet
        related data plane functions as decomposed into two sub-layers: a service
        sub-layer and a forwarding sub-layer.
       </t>

        <t>
        <xref target="ProtStack1"/> reproduced from the  <xref target="I-D.ietf-detnet-architecture"/>,shows a logical DetNet service with the two
        sub-layers.

       </t>
        <figure align="center" anchor="ProtStack1"
        title="DetNet data plane protocol stack">
        <artwork align="center"><![CDATA[
   |  packets going  |        ^  packets coming   ^
   v down the stack  v        |   up the stack    |
+-----------------------+   +-----------------------+
|        Source         |   |      Destination      |
+-----------------------+   +-----------------------+
|   Service sub-layer:  |   |   Service sub-layer:  |
|   Packet sequencing   |   | Duplicate elimination |
|    Flow replication   |   |      Flow merging     |
|    Packet encoding    |   |    Packet decoding    |
+-----------------------+   +-----------------------+
| Forwarding sub-layer: |   | Forwarding sub-layer: |
|  Resource allocation  |   |  Resource allocation  |
|    Explicit routes    |   |    Explicit routes    |
+-----------------------+   +-----------------------+
|     Lower layers      |   |     Lower layers      |
+-----------------------+   +-----------------------+
            v                           ^
             \_________________________/
        ]]></artwork>
        </figure>
       <t>
         The DetNet forwarding sub-layer may be directly provided by the DetNet
	 service sub-layer, for example by IP tunnels or MPLS. 
	 Alternatively, an overlay approach
         may be used in which the packet is natively carried between key nodes
         within the DetNet network (say between PREOF nodes) and a sub-layer is
         used to provide the information needed to reach the next hop in the
         overlay.
       </t>
       <t>
         The forwarding sub-layer provides the QoS related functions needed by the
         DetNet flow. It may do this directly through the use of
         queuing techniques and traffic engineering methods, or it may do this
         through the assistance of its underlying connectivity. For example it
         may call upon Ethernet TSN capabilities defined in IEEE 802.1 TSN
         <xref target="IEEE802.1TSNTG"/>.
       </t>
       <t>
         The service sub-layer provides additional support beyond the
         connectivity function of the forwarding sub-layer. An example
         of this is Packet Replication, Elimination, and Ordering
         functions see <xref target="PREOF"/>.
       </t>
       <t>
         The method of instantiating each of the layers is specific to the 
		 particular DetNet data plane method, and more than one approach may 
		 be applicable to a given bearer network type.
       </t>

      <section title="Data Plane Characteristics" anchor="sec_dp_char">
        <t>
	There are two major characteristics to the data plane: the technology and 
	the encapsulation, as discussed below.
        </t>
		<section title="Data Plane Technology" anchor="sec_dp_tech">
         <t>
         The DetNet data plane is provided by the DetNet service and forwarding 
		 sub layers. 
		 The DetNet service sub-layer
	 generally provides its functions for the DetNet application flows by using or applying
	 existing standardized headers and/or encapsulations.  The Detnet
	 forwarding sub-layer may provide capabilities leveraging that same
	 header or encapsulation technology (e.g., DN IP or DN MPLS) or it
	 may be achieved by other technologies (e.g.,  <xref
		 target="dn_svc_encaps"/>).
         DetNet is currently defined for operation over packet switched (IP)
          networks or label switched (MPLS) networks.  
         </t>
		</section>
		<section title="Data Plane Format" anchor="sec_dp_form">
         <t>
          DetNet encodes specific flow attributes
	  (flow identity and sequence number) in packets. 
	  For example, in DetNet IP, 
          zero encapsulation is used and no sequence number
          is available, and in DetNet MPLS, DetNet specific information may be added
          explicitly to the packets in the format of S-label and d-CW.  
         </t>
		</section>

     </section>

      <section title="Encapsulation" anchor="sec_dp_encap">
         <t>
		 The encapsulation of a DetNet flow allows it to be sent over a   
		 data plane technology other than its native type. 		 
		 DetNet uses header information to perform traffic classification, 
		 i.e., identify DetNet flows, and provide DetNet service and 
		 forwarding functions. As mentioned above, DetNet may add headers, 
		 as is the case for DN MPLS, or may use headers that are already 
		 present, as is the case in DN IP.		 
	   <xref target="dn_svc_encaps"/> illustrates some relationships 
	   between the components.
         </t>

        <figure anchor="dn_svc_encaps" align="center"
        title="DetNet Service Examples">
         <artwork align="center"><![CDATA[

                          +-----+
                          | TSN |
     +-------+          +-+-----+-+
     | DN IP |          | DN MPLS |
  +--+--+----+----+   +-+---+-----+-+
  | TSN | DN MPLS |   | TSN | DN IP |
  +-----+---------+   +-----+-------+

    ]]></artwork>
        </figure>

         <t>
           The use of encapsulation is also required if additional information
           (metadata) is needed by the DetNet data plane and there is either
           no ability to include it in the client data packet, or the
           specification of the client data plane does not permit the
           modification of the packet to include additional data. An example of
           such metadata is the inclusion of a sequence number required by the
           PREOF function.
         </t>
         <t>
           Encapsulation may also be used to carry or aggregate flows for equipment with limited
	   DetNet capability.
          </t>
      </section>
      <section title="DetNet Specific Metadata" anchor="sec_dp_metadata">
         <t>
          The DetNet data plane can provide or carry metadata: 
          <list style="numbers">
          <t>   
            Flow-ID
          </t>
          <t>   
            Sequence Number
          </t>
          </list>
          </t>
          <t>
		  The DetNet data plane framework supports a Flow-ID (for identification of the 
		  flow or aggregate flow) and/or a Sequence Number (for PREOF) for each 
		  DetNet flow. The DetNet Service sub-layer requires both; the DetNet 
		  forwarding sub-layer requires only Flow-ID. Metadata can also be 
		  used for OAM indications and instrumentation of DetNet data plane 
		  operation.
          </t>
          <t>
		  Metadata can be included implicit or explicit. Explicit means that 
		  a dedicated header field is used to include metadata in a DetNet 
		  packet. In case of implicit method a part of an already existing 
		  header field is used to encode the metadata.
          </t>
          <t>
          Explicit inclusion of metadata is possible through the use of
          IP options or IP extension headers. New IP options are almost impossible
          to get standardized or to deploy in an operational network and will
          not be discussed further in this text. IPv6 extensions headers are
          finding popularity in current IPv6 development work, particularly
          in connection with Segment Routing of IPv6 (SRv6) and IP OAM. The design
          of a new IPv6 extension header or the modification of an existing one is
          a technique available in the tool box of the DetNet IP data plane designer.
        </t>
        <t>
          Explicit inclusion of metadata in an IP packet is also possible
          through the inclusion of an MPLS label stack and the MPLS DetNet
          Control Word using one of the methods for carrying MPLS over IP <xref
                  target="I-D.ietf-detnet-mpls-over-udp-ip"/>. This is described in more
          detail in <xref target="subnet_considerations"/>.
        </t>
        <t>
          Implicit metadata in IP can be included through the use of the network
          programming paradigm <xref
                  target="I-D.ietf-spring-srv6-network-programming"/> in which the
          suffix of an IPv6 address is used to encode additional information
          for use by the network of the receiving host. 
        </t>
        <t>
          Some MPLS examples of implicit metadata include the sequence number for use by
          the PREOF function, or even all the essential information being
          included into the DetNet over MPLS label stack (the DetNet Control
          Word and the DetNet Service label).  
         </t>

      </section>

      <section title="DetNet IP Data Plane" anchor="sec_dt_ip_dp">
        <t>
          An IP data plane may operate natively or through the use of an
          encapsulation. 
          Many types of IP encapsulation can satisfy DetNet requirements 
          and it is anticipated that more than one encapsulation
          may be deployed, for example GRE, IPSec etc.
        </t>
        <t>
          One method of operating an IP DetNet data plane without encapsulation
          is to use "6-tuple" based flow identification, where "6-tuple" refers
          to information carried in IP and higher layer protocol headers.
          General background on the use of IP headers, and "6-tuples", to
          identify flows and support Quality of Service (QoS) can be found in
          <xref target="RFC3670"/>.  <xref target="RFC7657"/> provides
          useful background on differentiated services (DiffServ)
          and "tuple" based flow identification.  DetNet flow aggregation may
          be enabled via the use of wildcards, masks, prefixes and ranges.  The
          operation of this method is described in detail in <xref
                  target="I-D.ietf-detnet-ip"/>.
        </t>
        <t>
	  The DetNet forwarding plane may use explicit route capabilities and
	  traffic engineering capabilities to provide a forwarding sub-layer
	  that is responsible for providing resource allocation and explicit
	  routes.  It is possible to include such information in a native IP packet
	  explicitly, or implicitly.
        </t>
    </section>
      <section title="DetNet MPLS Data Plane" anchor="sec_dt_mpls_dp">
        <t>
          MPLS provides the ability to forward traffic over implicit
          and explicit paths to the point in the network where the next
          DetNet service sub-layer action needs to take place. It does this
          through the use of a stack of one or more labels with various
          forwarding semantics.
        </t>
        <t>
          MPLS also provides the ability to identify a service instance
          that is used to process the packet through the use of a label that
          maps the packet to a service instance.
        </t>
        <t>
          In cases where metadata is needed to process an MPLS encapsulated
          packet at the service sub-layer, a shim layer called a control
          word (CW)  <xref target="RFC4385"/> can be used.  Although such CWs
          are frequently 32 bits long, there is no architectural constraint on
          its size of this structure, only the requirement that it is fully
          understood by all parties operating on it in the DetNet service
          sub-layer.  The operation of this method is described in detail in
          <xref target="I-D.ietf-detnet-mpls"/>.
        </t>
      </section>

<section title="Further DetNet Data Plane Considerations" anchor="further_dn_dp_consid">
  <t>
    This section provides informative considerations related to
    providing DetNet service to flows which are identified
    based on their header information. 
  </t>
  <section title="Per Flow Related Functions" anchor="further_dn_dp_pf">
  <t>
	At a high level, the following functions are provided on a per flow basis.
  </t>
  <section title="Reservation and Allocation of resources">
      <t>
        Reservation of resources can allocate resources to specific DetNet
        flows.  This can eliminate packet contention and packet loss for DetNet
        traffic. This also can reduce jitter for DetNet traffic.  Resources 
		allocated to a DetNet flow protect it from other traffic flows. On the 
		other hand, DetNet flows are assumed to behave with respect to the reserved traffic
        profile. Misbehaving DetNet flows must be detected and it have to 
		be ensured that they do not compromise QoS of other flows. The use of
        (queuing, policing, shaping) policies can be used to ensure that the
        allocation of resources reserved for DetNet is met. 
        </t>
  </section>
  <section title="Explicit routes">
      <t>
        Use of a specific path for a flow. This allows control of the network
        delay by steering the packet with the ability to influence the physical
        path. Explicit routes complement reservation by ensuring that a
        consistent path can be associated with its resources for the duration
        of that path.  Coupled with the traffic mechanism, this limits
        misordering and bounds latency. Explicit route computation can
        encompass a wide set of constraints and optimize the path for a certain
        characteristic e.g. highest bandwidth or lowest jitter.  In these cases
        the "best" path for any set of characteristics may not be a shortest
        path.  The selection of path can take into account multiple network
        metrics. Some of these metrics are measured and distributed by the
        routing system as traffic engineering metrics.
        </t>
  </section>
  <section title="Service protection">
      <t>
        Use of multiple packet streams using multiple paths, for example 1+1 or
        1:1 linear protection.  For DetNet this primarily relates to packet
        replication and elimination capabilities. 
        MPLS offers a number of protection schemes. 
        MPLS hitless protection can be used to switch traffic to an already established path 
        in order to restore delivery rapidly after a failure. 
        Path changes, even in the case of failure recovery, can lead to the out of
        order delivery of data requiring packet ordering functions either
        within the DetNet service or at a high layer in the application
        traffic.  Establishment of new paths after a failure is out of scope
        for DetNet services.
        </t>
  </section>
  <section title="Network Coding">
      <t>
	Network Coding, not to be confused with network programming,
	comprises
        several techniques where multiple data flows are encoded. These
        resulting flows can then be sent on different paths. The encoding
        operation can combine flows and error recovery information.  When the
        encoded flows are decoded and recombined the original flows can be
        recovered.  Note that Network coding uses an alternative to packet by
        packet PREOF. Therefore, for certain network topologies and traffic
        loads, Network Coding can be used to improve a network's throughput,
        efficiency, latency, and scalability, as well as resilience to
        partition, attacks, and eavesdropping, as compared to traditional
        methods. DetNet could utilized Network coding as an alternative to
        other protection means.  Network coding is often applied in wireless
        networks and is being explored for other network types.
      </t>
  </section>
  <section title="Load sharing">
      <t>
        Use of packet-by-packet distribution of the same DetNet flow over
        multiple paths is not recommended except for the cases listed above
        where PREOF is utilized to improve protection of traffic and maintain order.
        Packet by packet load sharing, e.g., via ECMP or UCMP, impacts ordering and
        possibly jitter.
      </t>
      <!-- I think this section below should refer to OAM for the various underlying
           technology the one area that may be for consideration is how detnet can identify
           specific tunneled traffic. Of course that allows for security vulnerabilities -->
  </section>
  <section title="Troubleshooting">
      <t>
      Detnet leverages many different forwarding sub-layers, each of which 
	  supports various tools to troubleshoot connectivity, for example 
	  identification of misbehaving flows. The DetNet Service layer can 
	  leverage existing mechanisms to troubleshoot or monitor flows, such as 
	  those in use by IP and MPLS networks.  At the Application layer a client 
	  of a DetNet service can use existing techniques to detect and monitor 
	  delay and loss.
      </t>
  </section>
  <section title="Flow recognition for analytics">
      <t>
      Network analytics can be inherited from the technologies of the Service 
	  and Forwarding sub-layers.  At the DetNet service edge, packet and bit 
	  counters (e.g. sent, received, dropped, and out-of-sequence) can be 
	  maintained.
      </t>
  </section>
  <section title="Correlation of events with flows">
      <t>
      The provider of a DetNet service may provide other capabilities to 
	  monitor flows, such as more detailed loss statistics and time stamping 
	  of events.  The details of these capabilities are currently out of scope 
	  for this document.
      </t>
  </section>
  </section> <!-- End of Per Flow Related Functions -->

  <section title="Service Protection">
   <t>
      Service protection allow DetNet services to increase reliability and
      maintain a DetNet Service Assurance in the case of network congestion or
      network failure.  Detnet relies on the underlying technology capabilities
      for various protection schemes.  Protection schemes enable partial or
      complete coverage of the network paths and active protection with
      combinations of PRF, PEF, and POF.
   </t>

     <section title="Linear Service Protection">
      <t>
        An example DetNet MPLS network fragment and packet flow is illustrated in
        <xref target="dn_protection_flow"/>.
      </t>

       <figure anchor="dn_protection_flow" align="center"
       title="Example Packet Flow in DetNet protected Network">
           <artwork align="center"><![CDATA[


   1      1.1       1.1      1.2.1    1.2.1      1.2.2
CE1----EN1--------R1-------R2-------R3--------EN2-----CE2
         \           1.2.1 /                   /
          \1.2     /-----+                   /
           +------R4------------------------+
                     1.2.2
     ]]></artwork>
         </figure>

      <t>
     In <xref target="dn_protection_flow"/> the numbers are used to identify the
     instance of a packet.  Packet 1 is the original packet, and packets 1.1, and
     1.2 are two first generation copies of packet 1.  Packet 1.2.1 is a second
     generation copy of packet 1.2 etc.  Note that these numbers never appear in
     the packet, and are not to be confused with sequence numbers, labels or any
     other identifier that appears in the packet.  They simply indicate the
     generation number of the original packet so that its passage through the
     network fragment can be identified to the reader.
     </t>
     <t>
     Customer Equipment CE1 sends a packet into the DetNet enabled network.  This
     is packet (1).  Edge Node EN1 encapsulates the packet as a DetNet Packet and
     sends it to Relay node R1 (packet 1.1).  EN1 makes a copy of the packet
     (1.2), encapsulates it and sends this copy to Relay node R4.
     </t>
     <t>
     Note that along the path from EN1 to R1 there may be zero or more nodes
     which, for clarity, are not shown.  The same is true for any other path
     between two DetNet entities shown in <xref target="dn_protection_flow"/> .
     </t>
     <t>
     Relay node R4 has been configured to send one copy of the packet to Relay
     Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 1.2.2).
     </t>
     <t>
     R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, having
     been configured to perform packet elimination on this DetNet flow, forwards
     packet 1.2.1 to Relay Node R3.  Packet copy 1.1 is of no further use and so
     is discarded by R2.
     </t>
     <t>
     Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives packet
     copy 1.2.1 from R2 via relay Node R3.  EN2 therefore strips any DetNet
     encapsulation from packet copy 1.2.2 and forwards the packet to CE2.  When
     EN2 receives the later packet copy 1.2.1 this is discarded.
     </t>
     <t>
     The above is of course illustrative of many network scenarios that can be
     configured. 
     </t>
     <t>
     This example also illustrates 1:1 protection scheme meaning there is
     traffic over each segment of the end to end path. Local DetNet
     relay nodes determine which packets are eliminated and which packets are
     forwarded.  A 1+1 scheme where only one path is used for traffic at a
     time, could use the same topology. In this case there is no PRF function
     and traffic is switched upon detection of failure.  An OAM scheme that
     monitors the paths detects the loss of path or traffic is required to
     initiate the switch.  A POF may still be used in this case to
     prevent misordering of packets. In both cases the protection paths are
     established and maintained for the duration of the DetNet service.
     </t>
   </section>
     <section title="Ring Service Protection">
     <t>
     Ring protection may also be supported if the underlying technology
     supports it. Many of the same concepts apply however rings are normally
     1+1 protection for data efficiency reasons. <xref target="RFC8227"/> is an
     example of MPLS-TP data plane that supports Ring protection.
     </t>
   </section>
 </section>

 <section title="Aggregation Considerations" anchor = "aggregation">
  <t>
  The DetNet data plane also allows for the aggregation of DetNet flows, which 
  can improve scalability by reducing the per-hop state.  How this is accomplished is data
  plane or control plane dependent.  When DetNet flows are aggregated, transit
  nodes provide service to the aggregate and not on a per-DetNet flow basis.
  When aggregating DetNet flows the flows should be compatible i.e. the same or
  very similar QoS and CoS characteristics.  In this case, nodes performing
  aggregation will ensure that per-flow service requirements are achieved.
 </t>
  <t>
  If bandwidth reservations are used, the sum of the reservations should be the 
  sum of all the individual reservations; in other words, the reservations 
  should not add up to an over-subscription of bandwidth reservation. If 
  maximum delay bounds are used, the system should ensure that the aggregate 
  does not exceed the delay bounds of the individual flows.
 </t>
 <t>
  <!-- DetNet encapsulation is a data plane mechanism that can be used to aggregate
  traffic.  Encapsulation can either be in the same service type or in a
  different service type see <xref target="dn_svc_encaps"/> for example.--> When
  an encapsulation is used the choice of reserving a maximum resource level and
  then tracking the services in the aggregated service or adjusting the
  aggregated resources as the services are added is implementation and
  technology specific.
 </t>
 <t>
  DetNet flows at edges must be able to handle rejection to an aggregation
  group due to lack of resources as well as conditions where 
  requirements are not satisfied.
 </t>
   <section title="IP Aggregation" anchor = "ip-agg">
   <t>
   IP aggregation has both data plane and controller plane aspects. For the
   data plane, flows may be aggregated for treatment based on shared
   characteristics such as 6-tuple.  Alternatively,  an IP encapsulation may be
   used to tunnel an aggregate number of DetNet Flows between relay nodes.
   </t>
   </section>
   <section title="MPLS Aggregation" anchor = "mpls-agg">
   <t>
   MPLS aggregation also has data plane and controller plane aspects.  MPLS 
   flows are often tunneled in a forwarding sub-layer, under the reservation 
   associated with that MPLS tunnel.
   </t>
   </section>
 </section>
 <section title="End-System-Specific Considerations">
  <t>
   Data-flows requiring DetNet service are generated and terminated on
   end-systems. Encapsulation depends on the application and its
   preferences. For example, in a DetNet MPLS domain the sub-layer functions use the d-CWs,
   S-Labels and F-Labels to provide DetNet services. However, an
   application may exchange further flow related parameters (e.g.,
   time-stamp), which are not provided by DetNet functions.
  </t>

   <t>
   As a general rule, DetNet domains are capable of forwarding any
   DetNet flows and the DetNet domain does not mandate the
   end-system or edge node encapsulation format. Unless there is a
   proxy of some form present, end-systems peer with similar end-systems
   using the same application encapsulation format.  For example, as
   shown in <xref target="fig_es_connect"/>, IP applications peer with
   IP applications and Ethernet applications peer with Ethernet
   applications.
  </t>

  <figure title="End-Systems and The DetNet MPLS Domain" anchor="fig_es_connect">
  <artwork align="center"><![CDATA[
+-----+
|  X  |                               +-----+
+-----+                               |  X  |
| Eth |               ________        +-----+
+-----+     _____    /        \       | Eth |
       \   /     \__/          \___   +-----+
        \ /                        \ /
         0======== tunnel-1 ========0_
         |                             \
          \                             |
          0========= tunnel-2 =========0
         / \                        __/ \
  +-----+   \__ DetNet MPLS domain /     \
  |  X  |      \         __       /       +-----+
  +-----+       \_______/  \_____/        |  X  |
  |  IP |                                 +-----+
  +-----+                                 |  IP |
                                          +-----+
  ]]>
    </artwork></figure>

 </section>
 <section title="Sub-Network Considerations" anchor="subnet_considerations">
   <t>
     Any of the DetNet service types may be transported by another
     DetNet service. MPLS nodes may interconnected by different sub-network
     technologies, which may include point-to-point links. Each of these
     sub-network technologies need to provide appropriate service to DetNet
     flows.  In some cases, e.g., on dedicated point-to-point links or TDM
     technologies, all that is required is for a DetNet node to appropriately
     queue its output traffic.  In other cases, DetNet nodes will need to map
     DetNet flows to the flow semantics (i.e., identifiers) and mechanisms used
     by an underlying sub-network technology.  <xref
             target="fig_dn_mpls_sn_ex"/> shows several examples of header
     formats that can be used to carry DetNet MPLS flows over different
     sub-network technologies.  L2 represent a generic layer-2 encapsulation
     that might be used on a point-to-point link.  TSN represents the
     encapsulation used on an IEEE 802.1 TSN network, as described in <xref
             target="I-D.ietf-detnet-mpls-over-tsn"/>.  UDP/IP represents the encapsulation
     used on a DetNet IP PSN, as described in <xref
             target="I-D.ietf-detnet-mpls-over-udp-ip"/>.
   </t>


  <figure title="Example DetNet MPLS Sub-Network Formats" anchor="fig_dn_mpls_sn_ex">
  <artwork align="center"><![CDATA[

                   +------+  +------+  +------+
App-Flow           |  X   |  |  X   |  |  X   |
             +-----+======+--+======+--+======+-----+
DetNet-MPLS        | d-CW |  | d-CW |  | d-CW |
                   +------+  +------+  +------+
                   |Labels|  |Labels|  |Labels|
             +-----+======+--+======+--+======+-----+
Sub-Network        |  L2  |  | TSN  |  | UDP  |
                   +------+  +------+  +------+
                                       |  IP  |
                                       +------+
                                       |  L2  |
                                       +------+
    ]]>
  </artwork></figure>
 </section>
</section>

</section>
<!-- ================================================================= -->

<section title="Controller Plane (Management and Control)
                Considerations" anchor="cp_considerations">
    <section title="DetNet Controller Plane Requirements"
             anchor="control-management-requirements">
      <t>
        While the definition of controller plane for DetNet is out of
        the scope of this document, there are particular
        considerations and requirements for such that result from the
        unique characteristics of the DetNet architecture <xref
        target="I-D.ietf-detnet-architecture"/> and data plane as
        defined herein.
      </t>
      <t>
        The primary requirements of the DetNet controller plane are
        that it must be able to:
        <list style="symbols">
          <t>
            Instantiate DetNet flows in a DetNet domain (which may
            include some or all of explicit path determination, link
            bandwidth reservations, restricting flows to IEEE 802.1 TSN
            links, node buffer and other resource reservations,
            specification of required queuing disciplines along the
            path, ability to manage bidirectional flows, etc.) as needed
            for a flow.
          </t>
          <t>
            In the case of MPLS, manage DetNet S-Label and F-Label allocation
            and distribution, where the DetNet MPLS encapsulation is in use see
            <xref target="I-D.ietf-detnet-mpls"/>.
          </t>
          <t>
            Support DetNet flow aggregation.
          </t>
          <t>
            Advertise static and dynamic node and link resources such
            as capabilities and adjacencies to other network nodes (for
            dynamic signaling approaches) or to network controllers (for
            centralized approaches).
          </t>
          <t>
            Scale to handle the number of DetNet flows expected in a
            domain (which may require per-flow signaling or
            provisioning).
          </t>
          <t>
            Provision flow identification information at each of the nodes
            along the path. Flow identification may differ depending on the
            location in the network and the DetNet functionality (e.g. transit
            node vs. relay node).
          </t>
        </list>
      </t>
      <t>
        These requirements, as stated earlier, could be satisfied using
        distributed control protocol signaling (such as RSVP-TE),
        centralized network management provisioning mechanisms (such as
        BGP, PCEP, YANG <xref
        target="I-D.ietf-detnet-flow-information-model"/>, etc.) or hybrid
        combinations of the two, and could also make use of MPLS-based
        segment routing.
      </t>
      <t>
        In the abstract, the results of either distributed signaling
        or centralized provisioning are equivalent from a DetNet data
        plane perspective - flows are instantiated, explicit routes
        are determined, resources are reserved, and packets are
        forwarded through the domain using the DetNet data plane.
      </t>
      <t>
        However, from a practical and implementation standpoint, they are not
        equivalent at all. Some approaches are more scalable than others in
        terms of signaling load on the network. Some can take advantage of
        global tracking of resources in the DetNet domain for better overall
        network resource optimization. Some are more resilient than others if
        link, node, or management equipment failures occur. While a detailed
        analysis of the control plane alternatives is out of the scope of this
        document, the requirements from this document can be used as the basis
        of a later analysis of the alternatives.
      </t>
    </section>

  <section title="Generic Controller Plane Considerations" anchor="gen_cp_considerations">
    <t>
      This section covers control plane considerations that are
      independent of the data plane technology used for DetNet service
      delivery.
    </t>
    <t>
      While management plane and control planes are traditionally
      considered separately, from the Data Plane perspective there is no
      practical difference based on the origin of flow provisioning
      information, and the DetNet architecture <xref
      target="I-D.ietf-detnet-architecture"/> refers to these collectively
      as the 'Controller Plane'.  This document therefore does not
      distinguish between information provided by distributed control
      plane protocols, e.g., RSVP-TE <xref target="RFC3209"/> and <xref
      target="RFC3473"/>, or by centralized network management mechanisms,
      e.g., RestConf <xref target="RFC8040"/>, YANG <xref
      target="RFC7950"/>, and the Path Computation Element Communication
      Protocol (PCEP) <xref
      target="I-D.ietf-pce-pcep-extension-for-pce-controller"/> or any
      combination thereof. Specific considerations and requirements for
      the DetNet Controller Plane are discussed in <xref
      target="control-management-requirements"/>.
    </t>
    <t>
      Each respective data plane document also covers the control plane considerations
      for that technology.  For example <xref target="I-D.ietf-detnet-ip"/> covers IP
      control plane normative considerations and  <xref
      target="I-D.ietf-detnet-mpls"/>  covers MPLS control plane
      normative considerations. 	
      </t>

    <section title="Flow Aggregation Control">
      <t>
        Flow aggregation means that multiple App-flows are served by a single new DetNet
        flow.  There are many techniques to achieve aggregation, for example in case of IP, 
		it can be grouping of IP flows that share 6-tuple attributes or flow 
		identifiers at the DetNet sub-layer.
		Another example includes aggregation accomplished through the use of
		hierarchical LSPs in MPLS and tunnels.
      </t>
       <t>
	Control of aggregation involves a set of procedures listed here.  
	Aggregation may use some or all of these capabilities and the order may vary:
       <list style="symbols">
	 <t> Traffic engineering resource collection and distribution:
           <list style="empty">
             <t>
	       Available resources are tracked through control plane or
	       management plane databases and distributed amongst controllers
	       or nodes that can manage resources.
             </t>
           </list>
         </t>
	 <t> Path computation and resource allocation:
           <list style="empty">
             <t>
	       When DetNet services are provisioned or requested one or more
	       paths meeting the requirements are selected and the resources
	       verified and recorded.
             </t>
           </list>
       </t>
       <t> Resource assignment and data plane co-ordination:
           <list style="empty">
             <t>
	       The assignment of resources along the path depends on the technology and it
	       includes assignment of specific links and coordination of the
	       queuing and other traffic management capabilities such as policing and shaping.
             </t>
           </list>
       </t>
         <t> Assigned Resource recording and updating:
           <list style="empty">
             <t>
	       Depending on the specific technology, the assigned resources are 
		   updated and distributed in the databases, preventing over-subscription.
             </t>
           </list>
       </t>
      </list>
      </t>

    </section>

    <section title="Explicit Routes">
      <t>
        Explicit routes are used to ensure that packets are routed through the
        resources that have been reserved for them, and hence provide the
        DetNet application with the required service.  A requirement for the
        DetNet Controller Plane will be the ability to assign a particular
        identified DetNet IP flow to a path through the DetNet domain that has
        been assigned the required nodal resources.  This provides the
        appropriate traffic treatment for the flow and also includes particular
        links as a part of the path that are able to support the DetNet flow.
	For example, by using IEEE 802.1 TSN links (as discussed in
	<xref target="I-D.ietf-detnet-mpls-over-tsn"/> ) DetNet parameters can be maintained.
        Further considerations and requirements for the DetNet Controller Plane
        are discussed in <xref target="control-management-requirements"/>.
      </t>
      <t>
        Whether configuring, calculating and instantiating these
        routes is a single-stage or multi-stage process, or in a
        centralized or distributed manner, is out of scope of this
        document.
      </t>
      <t>
        There are several approaches that could be used to provide
        explicit routes and resource allocation in the DetNet
        forwarding sub-layer. For example:
        <list style="symbols">
          <t>
            The path could be explicitly set up by a controller which
            calculates the path and explicitly configures each node along that
            path with the appropriate forwarding and resource allocation
            information.
          </t>
          <t>
            The path could use a distributed control plane such as
            <xref target="RFC2205">RSVP</xref> or RSVP-TE <xref
            target="RFC3473"/> extended to support DetNet IP flows.
          </t>
          <t>
            The path could be implemented using IPv6-based segment
            routing when extended to support resource allocation.
          </t>
        </list>
        See <xref target="control-management-requirements"/> for
        further discussion of these alternatives. In addition, <xref
        target="RFC2386"/> contains useful background information on
        QoS-based routing, and <xref target="RFC5575"/> discusses a
        specific mechanism used by BGP for traffic flow specification
        and policy-based routing.
      </t>
    </section>
    <section title="Contention Loss and Jitter Reduction">
      <t>
        As discussed in <xref target="sec_intro"/>, this document does
        not specify the mechanisms needed to eliminate packet contention, packet loss
        or reduce jitter for DetNet flows at the DetNet forwarding
        sub-layer.  The ability to manage node and link resources to
        be able to provide these functions is a necessary part of
        the DetNet controller plane. It is also necessary to be
        able to control the required queuing mechanisms used to
        provide these functions along a flow's path through the
        network. See  <xref target="I-D.ietf-detnet-ip"/> and <xref
        target="control-management-requirements"/> for further
        discussion of these requirements.
      </t>
    </section>
    <section title="Bidirectional Traffic">
      <t>
        DetNet applications typically generate bidirectional traffic. IP and
        MPLS typically treat each direction separately and do not force
        interdependence of each direction.  MPLS has considered bidirectional
        traffic requirements and the MPLS definitions from <xref
                target="RFC5654"/> are useful to illustrate terms such as
        associated bidirectional flows and co-routed bidirectional flows.  MPLS
        defines a point-to-point associated bidirectional LSP as consisting of
        two unidirectional point-to-point LSPs, one from A to B and the other
        from B to A, which are regarded as providing a single logical
        bidirectional forwarding path.  This is analogous to standard IP
        routing.  MPLS defines a point-to-point co-routed bidirectional LSP as
        an associated bidirectional LSP which satisfies the additional
        constraint that its two unidirectional component LSPs follow the same
        path (in terms of both nodes and links) in both directions.  An
        important property of co-routed bidirectional LSPs is that their
        unidirectional component LSPs share fate.  In both types of
        bidirectional LSPs, resource reservations may differ in each direction.
        The concepts of associated bidirectional flows and co-routed
        bidirectional flows can also be applied to DetNet IP flows.
      </t>
      <t>
        While the DetNet IP data plane must support bidirectional
        DetNet flows, there are no special bidirectional features with
        respect to the data plane other than the need for the two
        directions of a co-routed bidirectional flow to take the same
        path. That is to say that bidirectional DetNet flows are
        solely represented at the management and control plane levels,
        without specific support or knowledge within the DetNet data
        plane.  Fate sharing and associated or co-routed
        bidirectional flows, can be managed at the control level.
        </t>
         <t>
        DetNet's use of PREOF may increase the complexity of using co-routing 
		bidirectional flows, since if PREOF is used, then the replication points 
		in one direction would have to match the elimination points in the 
		other direction, and vice versa. In such cases the optimal points for 
		these functions in one direction may not match the optimal points in 
		the other, due to network and traffic constraints.
        Furthermore, due to the per packet service protection nature, 
        bidirectional forwarding per packet may not be ensured. 
        The first packet of received member flows is selected by the 
        elimination function independently of which path it has taken 
        through the network.
        </t>
        <t>
        Control and management mechanisms need to support bidirectional flows,
        but the specification of such mechanisms are out of scope of this
        document. An example control plane solution for MPLS can be found in <xref
        target="RFC3473"/> , <xref target="RFC6387"/> and <xref
                target="RFC7551"/>.  These requirements are included in <xref
        target="control-management-requirements"/>.
      </t>
<!--
    -->
    </section> 
  </section>

    <section title="Packet Replication, Elimination, and Ordering (PREOF)" anchor="PREOF">
      <t>
        The controller plane protocol solution required for managing the
        PREOF processing is outside the scope of this document. That
        said, it should be noted that the ability to determine, for a
        particular flow, optimal packet replication and elimination
        points in the DetNet domain requires explicit support. There may
        be capabilities that can be used, or extended, for example GMPLS
        end-to-end recovery <xref target="RFC4872"/> and GMPLS segment
        recovery <xref target="RFC4873"/>.
      </t>
    </section>
 </section>

    <!-- ===================================================================== -->
<section title="Security Considerations">
  <t>
    Security considerations for DetNet are described in detail in
    <xref target="I-D.ietf-detnet-security"/>. General security considerations
    are described in <xref target="I-D.ietf-detnet-architecture"/>. This section
    considers general security considerations applicable to all data planes.
   </t>
   <t>
    Security aspects which are unique to DetNet are those whose aim is to
    provide the specific quality of service aspects of DetNet, which are
    primarily to deliver data flows with extremely low packet loss rates
    and bounded end-to-end delivery latency.
     </t>
     <t>
     The primary considerations for the data plane is to maintain
     integrity of data and delivery of the associated DetNet service traversing
     the DetNet network.
     Application flows can be protected through whatever means is
     provided by the underlying technology. For example, encryption may be
     used, such as that provided by IPSec <xref target="RFC4301"/> for IP
     flows and/or by an underlying sub-net using MACSec
     <xref target="IEEE802.1AE-2018"/> for Ethernet (Layer-2) flows.
     </t>
     <t>
     At the management and control level DetNet flows are identified on a
     per-flow basis, which may provide controller plane
     attackers with additional information about the data flows (when
     compared to controller planes that do not include per-flow identification).
     This is an inherent property of DetNet which has security
     implications that should be considered when determining if DetNet is
     a suitable technology for any given use case.
     </t>
     <t>
     To provide uninterrupted availability of the DetNet
     service, provisions can be made against DOS attacks and delay
     attacks. To protect against DOS attacks, excess traffic due to
     malicious or malfunctioning devices can be prevented or mitigated,
     for example through the use of existing mechanism such as policing and shaping applied at
     the input of a DetNet domain. To prevent DetNet packets from
     being delayed by an entity external to a DetNet domain, DetNet
     technology definition can allow for the mitigation of
     Man-In-The-Middle attacks, for example through use of
     authentication and authorization of devices within the DetNet domain.
     </t>
     <t>
	 In order to prevent or mitigate DetNet attacks on other networks via 
	 flow escape, edge devices can for example use existing mechanism such 
	 as policing and shaping applied at the output of a DetNet domain.
     </t>
</section>


<section anchor="iana" title="IANA Considerations">
  <t>
   This document makes no IANA requests.
  </t>
</section>

<section anchor="acks" title="Acknowledgements">
  <t>
    The authors wish to thank Pat Thaler, Norman Finn, Loa Anderson, David
    Black, Rodney Cummings, Ethan Grossman, Tal Mizrahi, David Mozes, Craig
    Gunther, George Swallow, Yuanlong Jiang and Carlos J. Bernardos for their
    various contributions to this work.  
  </t>
</section>

</middle>

<back>
  <references title="Normative References">
   <?rfc include="reference.RFC.3209"?>
   <?rfc include="reference.RFC.3473"?>
   <?rfc include="reference.RFC.4385"?>
  </references>
  <references title="Informative References">
   <?rfc include="reference.RFC.2205"?>
   <?rfc include="reference.RFC.2386"?>
   <?rfc include="reference.RFC.4872"?>
   <?rfc include="reference.RFC.4873"?>
   <?rfc include="reference.RFC.7657"?>
   <?rfc include="reference.RFC.3670"?>
   <?rfc include="reference.RFC.5575"?>
   <?rfc include="reference.RFC.5654"?>
   <?rfc include="reference.RFC.6387"?>
   <?rfc include="reference.RFC.7950"?>
   <?rfc include="reference.RFC.7551"?>
   <?rfc include="reference.RFC.8040"?>
   <?rfc include="reference.RFC.8227"?>
   <?rfc include="reference.RFC.4301"?>
   <?rfc include="reference.I-D.ietf-detnet-architecture"?>
   <?rfc include="reference.I-D.ietf-pce-pcep-extension-for-pce-controller"?>
   <?rfc include="reference.I-D.ietf-detnet-flow-information-model"?>
   <?rfc include="reference.I-D.ietf-detnet-ip"?>
   <?rfc include="reference.I-D.ietf-detnet-mpls"?>
   <?rfc include="reference.I-D.ietf-detnet-mpls-over-tsn"?>
   <?rfc include="reference.I-D.ietf-detnet-mpls-over-udp-ip"?>
   <?rfc include="reference.I-D.ietf-detnet-security"?>

   <!--reference anchor='I-D.ietf-detnet-ip-over-tsn'>
     <front>
       <title>DetNet Data Plane: IP over IEEE 802.1 Time Sensitive Networking (TSN)
       </title>
       <author>
         <organization>Korhonen, J., Varga, B.</organization>
       </author>
       <date year='2018' />
     </front>
   </reference-->

   <?rfc include="reference.I-D.ietf-spring-srv6-network-programming"?>
   
   <!--reference anchor="IEEE8021Q"
     target="http://standards.ieee.org/about/get/">
    <front>
      <title>Standard for Local and metropolitan area networks-Bridges and Bridged Networks (IEEE Std 802.1Q-2014)
      </title>
     <author>
      <organization>IEEE 802.1</organization>
     </author>
     <date year="2014"/>
    </front>
    <format type="PDF" target="http://standards.ieee.org/about/get/"/>
   </reference-->
   <reference anchor="IEEE802.1TSNTG" target="http://www.ieee802.org/1/tsn">
    <front>
     <title>IEEE 802.1 Time-Sensitive Networking Task Group</title>
     <author>
       <organization>IEEE Standards Association</organization>
       </author>
       <date></date>
       </front>
   </reference>
    <reference anchor="IEEE802.1AE-2018"
      target="https://ieeexplore.ieee.org/document/8585421">
      <front>
        <title>IEEE Std 802.1AE-2018 MAC Security (MACsec)</title>
        <author>
          <organization>IEEE Standards Association</organization>
        </author>
        <date year="2018" />
      </front>
    </reference>
   <!--reference anchor="IEEE8021CB"
    target="http://www.ieee802.org/1/files/private/cb-drafts/d2/802-1CB-d2-1.pdf">
    <front>
        <title>Draft Standard for Local and metropolitan area networks - Seamless Redundancy</title>
        <author initials="N. F." surname="Finn" fullname="Norman Finn">
            <organization>IEEE 802.1</organization>
        </author>
        <date month="December" year="2015"/>
    </front>
    <seriesInfo name="IEEE P802.1CB /D2.1" value="P802.1CB"/>
    <format type="PDF" target="http://www.ieee802.org/1/files/private/cb-drafts/d2/802-1CB-d2-1.pdf"/>
   </reference-->

   <!--reference anchor="G.8275.1"
              target="https://www.itu.int/rec/T-REC-G.8275.1/en">
     <front>
       <title>Precision time protocol telecom profile for phase/time synchronization with full timing support from the network</title>
       <author>
         <organization>International Telecommunication Union</organization>
       </author>
       <date month="June" year="2016"/>
     </front>
     <seriesInfo name="ITU-T G.8275.1/Y.1369.1" value="G.8275.1"/>
   </reference-->

   <!--reference anchor="G.8275.2"
              target="https://www.itu.int/rec/T-REC-G.8275.2/en">
     <front>
       <title>Precision time protocol telecom profile for phase/time synchronization with partial timing support from the network</title>
       <author>
         <organization>International Telecommunication Union</organization>
       </author>
       <date month="June" year="2016"/>
     </front>
     <seriesInfo name="ITU-T G.8275.2/Y.1369.2" value="G.8275.2"/>
   </reference-->

  </references>

 </back>
</rfc>
