Hi Michael,

Thank you very much for the thorough review and good comments!

Please find below in-line responses and how we plan to update the draft 
to resolve your comments.

Best regards,
Janos


On 9/29/2018 12:24 AM, Michael Scharf wrote:
> Reviewer: Michael Scharf
> Review result: Ready with Issues
>
> The document "Deterministic Networking Architecture"
> (draft-ietf-detnet-architecture-08) defines an overall framework for
> Deterministic Networking.
>
> As TSV-ART reviewer, I believe that this document has issues as 
> detailed below.
>
> Michael
>
> Major issues:
>
> * It seems that DetNet cannot easily be deployed in the Internet without
> additional means. Thus, for a baseline document, one could expect some
> explanation on the requirements of deploying DetNet in a network.

DetNet is not for the Internet. You are right, it should be clarified.
I suggest to extend the fist sentence of the Abstract:

OLD:
This document provides the overall Architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of
data flows with extremely low packet loss rates and bounded end-to-
end delivery latency.

NEW:
This document provides the overall Architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of
data flows with extremely low packet loss rates and bounded end-to-
end delivery latency within a network domain.


Change the first paragraph of the Introduction:

OLD:
This document provides the overall Architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of
data flows with extremely low packet loss rates and bounded end-to-
end delivery latency. DetNet operates at the IP layer and delivers
service over sub-network technologies such as MPLS and IEEE 802.1
TSN. DetNet accomplishes these goals by dedicating network resources
such as link bandwidth and buffer space to DetNet flows and/or
classes of DetNet flows, and by replicating packets along multiple
paths. Unused reserved resources are available to non-DetNet
packets.


NEW:
This document provides the overall Architecture for Deterministic
Networking (DetNet), which provides a capability for the delivery of
data flows with extremely low packet loss rates and bounded end-to-
end delivery latency within a network domain. DetNet is for networks
that are under a single administrative control or within a closed group
of administrative control; these include campus-wide networks and
private WANs. DetNet is not for large groups of domains such as the
Internet.

DetNet operates at the IP layer and delivers
service over sub-network technologies such as MPLS and IEEE 802.1
TSN. DetNet accomplishes these goals by dedicating network resources
such as link bandwidth and buffer space to DetNet flows and/or
classes of DetNet flows, and by replicating packets along multiple
paths. Unused reserved resources are available to non-DetNet
packets.

> DetNet
> basically requires support in (almost) all network devices 
> transporting DetNet
> traffic. That assumption should be explicitly spelt out early in the 
> document,
> e.g., in the introduction.

Actually, some form of DetNet support is required from each node that 
transports DetNet traffic. For instance, DetNet flows have to be 
recognized in order not to spoil their QoS at the minimum.

The Introduction could be extended to clarify, e.g., the third paragraph:

OLD:
A goal of DetNet is a converged network in all respects. That is,
the presence of DetNet flows does not preclude non-DetNet flows, and
the benefits offered DetNet flows should not, except in extreme
cases, prevent existing QoS mechanisms from operating in a normal
fashion, subject to the bandwidth required for the DetNet flows. A
single source-destination pair can trade both DetNet and non-DetNet
flows. End systems and applications need not instantiate special
interfaces for DetNet flows. Networks are not restricted to certain
topologies; connectivity is not restricted. Any application that
generates a data flow that can be usefully characterized as having a
maximum bandwidth should be able to take advantage of DetNet, as long
as the necessary resources can be reserved. Reservations can be made
by the application itself, via network management, by an
application’s controller, or by other means, e.g., a dynamic control
plane (e.g., [RFC2205]).

NEW:
A goal of DetNet is a converged network in all respects. That is,
the presence of DetNet flows does not preclude non-DetNet flows, and
the benefits offered DetNet flows should not, except in extreme
cases, prevent existing QoS mechanisms from operating in a normal
fashion, subject to the bandwidth required for the DetNet flows. A
single source-destination pair can trade both DetNet and non-DetNet
flows. End systems and applications need not instantiate special
interfaces for DetNet flows. Networks are not restricted to certain
topologies; connectivity is not restricted. Any application that
generates a data flow that can be usefully characterized as having a
maximum bandwidth should be able to take advantage of DetNet, as long
as the necessary resources can be reserved. Reservations can be made
by the application itself, via network management, by an
application’s controller, or by other means, e.g., a dynamic control
plane (e.g., [RFC2205]). Network nodes supporting DetNet flows have
to implement some of the DetNet capabilities (not necessarily all) in
order to treat DetNet flows such that their QoS requirements are met.


> There also needs to be an explicit discussion of the
> implications if not the whole network is aware of or supports DetNet. 
> There is
> some text in Section 4.2.2 and Section 4.3.3, but I believe additional 
> explicit
> discussion is needed at a prominant place. For instance, can use of 
> DetNet do
> harm to parts of a network not supporting DetNet? As a side note, when 
> TCPM
> published RFC 8257, the following disclaimer was added: "DCTCP, as 
> described in
> this specification, is applicable to deployments in controlled 
> environments
> like data centers, but it must not be deployed over the public 
> Internet without
> additional measures." I wonder if a similar disclaimer is needed for 
> DetNet. If
> there is an implicit assumption that DetNet will be used in homogenous
> environments with mostly DetNet-aware devices within the same 
> organization,
> such an assumption should be made explicit.
I think the suggested changes to the Abstract and the Introduction make 
it clear.


> * It is surprising that there is hardly any discussion on network 
> robustness
> and safety; this probably also relates to security. For instance,
> misconfiguration or errors of functions performing packet replication 
> could
> severely and permantly congest a network and cause harm. How does the 
> DetNet
> architecture ensure that a network stays fully operational e.g. if the 
> topology
> changes or there are equipment failures? Probably this can be solved by
> implementations (e.g., dynamic control plane), but why are corresponding
> requirements not spelt out? Section 3.3.2 speculates that filters and 
> policers
> can help, and that may be true, but that probably still assumes 
> consistently
> and correctly configured (and well-behaving) devices. And Section 3.3.2 is
> vague and mentions a "infinite variety of possible failures" without 
> stating
> any requirements or recommendations. There may be further solutions, 
> such as
> circuit breakers and the like. Why are such topics not discussed?
Security issues and considerations are addressed by the DetNet Security 
draft: https://datatracker.ietf.org/doc/draft-ietf-detnet-security.
Reference to the  DetNet Security draft will be added.

There will be a document dedicated to Operations, Administration and 
Maintenance (OAM), which will address operational, misconfiguration, and 
failure detection issues.

The topology changes have to be solved by the control plane, either 
centralized or distributed.

The filters and policers described in Section 3.3.2 also provide 
robustness by eliminating/mitigating traffic coming from malfunctioning 
devices.

RFC2475 is recommended for traffic policing functions to increase 
robustness.

The text  in Section 3.3.2 could be made clearer, e.g., by updating the 
first paragraph to:

OLD:
One key to building robust real-time systems is to reduce the
infinite variety of possible failures to a number that can be
analyzed with reasonable confidence. DetNet aids in the process by
allowing for filters and policers to detect DetNet packets received
on the wrong interface, or at the wrong time, or in too great a
volume, and to then take actions such as discarding the offending
packet, shutting down the offending DetNet flow, or shutting down the
offending interface.

NEW:
Robust real-time systems require to reduce the number of possible
failures. Filters and policers should be used in a DetNet network to
detect if DetNet packets are received on the wrong interface, or at
the wrong time, or in too great volume. Furthermore, filters and
policers can take action to discard offending packets or flows, or
trigger shutting down the offending DetNet flow, or shutting down
the offending interface.


> * Somewhat related, the document only looks at impact of failures to 
> the QoS of
> DetNet traffic. What is missing is a discussion how to protect 
> non-DetNet parts
> of a network from any harm caused by DetNet mechanisms. Solutions to this
> probably exist. But why is the impact on non-DetNet traffic (e.g., in 
> case of
> topology changes or failures of DetNet functions) not discussed at all 
> in the
> document?
Actually the regulation of DetNet traffic by polices and filters 
protects both other DetNet traffic and non-DetNet traffic.

Section 3.3.1 could be extended to make it clear, e.g., by extending the 
last paragraph:

OLD:
Ideally, the net effect of the presence of DetNet flows in a network
on the non-DetNet packets is primarily a reduction in the available
bandwidth.

NEW:
Traffic policing functions (e.g. [RFC2475]) are used to avoid the
starvation of non-DetNet traffic. Thus, the net effect of the presence
of DetNet flows in a network on non-DetNet flows is primarily a
reduction in the available bandwidth.

> * Regarding security, an architecture like DetNet probably requires 
> that only
> authenticated and authorized end systems have access to the data 
> plane. The
> security considerations only briefly mention the control aspect ("the
> authentication and authorization of the controlling systems").
Security issues and considerations are addressed by the DetNet Security 
draft: https://datatracker.ietf.org/doc/draft-ietf-detnet-security.
Reference to the DetNet Security draft will be added.


> * For an architecture document, the lack of clarity and consistency 
> regarding
> terminology is concerning. This specifically applies to the case of 
> incomplete
> networks (as per Section 4.2.2 and 4.3.3) that include "DetNet-unaware 
> nodes".
> The document introduces terms such as "DetNet intermediate nodes" but then
> repeatedly uses generic terms such as "node" or "hop" that may include
> DetNet-unaware nodes. For instance, for incomplete networks, a 
> sentence such as
> "The primary means by which DetNet achieves its QoS assurances is to 
> reduce, or
> even completely eliminate, congestion within a node as a cause of 
> packet loss"
> seems to only apply to "DetNet transit nodes" but not "DetNet-unaware 
> nodes".
> Similar ambiguity exist for other use of the terms "hop" and "node", 
> which may
> or may not include DetNet-unaware nodes. It is unclear why the 
> document does
> not consistently use the terminology introduced in Section 2.1 in all 
> sections
> and clearly distinguishes cases with and without DetNet support.
Each occurrence of "node" will be checked and updated to make it clear 
what type of node is meant.

Each occurrence of "hop" will be double checked and updated as necessary 
and possible to make it clear, preferably replaced with the appropriate 
type of node. Note that, for instance, "Per-Hop Behavior" cannot be changed.


> * Section 4.4 refers to RFC 7426, which is an informational RFC on 
> IRTF stream,
> and the document uses the concepts introduced there (e.g., "planes"). 
> This is
> very confusing. First, an IETF Proposed Standard should probably refer to
> documents having IETF consensus. An example would be RFC 7491, albeit 
> there is
> other related work as well, e.g., in the TEAS WG. Second, Section 4.4 
> is by and
> large decoupled from the rest of the document and not specific to DetNet.
> Neither do other sections of the document refer to the concepts 
> introduced in
> Section 4.4, nor does Section 4.4 use the DetNet terminology or discuss
> applicability to DetNet. Section 4.4 even mentions explicitly at the 
> end that
> it discusses aspects that are orthogonal to the DetNet architecture. 
> It is not
> at all clear why Section 4.4 is in this document. Section 4.4 could be 
> removed
> from the document without impacting the rest of the document.
The document should point out to traffic engineering and centralized 
control plane aspects, so it is better to keep Section 4.4.

RFC 7426 provides the full picture of SDN architecture, which needs to 
be referred from the DetNet architecture. No other RFC provides such 
detailed picture. Therefore, we need to keep the reference to RFC 7426, 
which is an informative reference.

The related work in TEAS is already referred to.

> Minor issues:
>
> * Terminology "DetNet transport layer"
>
> The term "transport layer" has a well-defined meaning in the IETF, e.g.
> originating from RFC 1122. While "transport" and e.g. "transport 
> network" is
> used in the IETF for different technologies in different areas, I 
> think the
> term "transport layer" is typically understood to refer to transport
> protocols such as TCP and UDP. As such, I personally find the term "DetNet
> transport layer" misleading and confusing. The confusion is easy to 
> see e.g.
> in Figure 4, where UDP (which is a transport protocol as per RFC 1122) 
> sits
> on top of "transport".
>
> Based on the document it also may be solution/implementation specific 
> whether
> the "DetNet transport layer" is actually a separate protocol layer 
> compared
> to the "DetNet service layer". Thus it is not clear to me why the word
> "layer" has to be used, specifically in combination "transport layer".
>
> To me as, the word "transport layer" (and "transport protocol") should be
> used for protocols defined in TSV area, consistent with RFC 1122. But 
> this is
> probably a question to be sorted out by the IESG.

"transport" is used here as in "packet transport networks" not as OSI L4 
transport.

I suggest to use the terms "DetNet transport sub-layer" and "DetNet 
service sub-layer" throughout the document, which would hopefully avoid 
confusion.

For instance, in 4.1.2 below Figure 3:

OLD:
Distinguishing the function of two DetNet data plane layers, the
DetNet service layer and the DetNet transport layer, helps to explore
and evaluate various combinations of the data plane solutions
available, some are illustrated in Figure 4. This separation of
DetNet layers, while helpful, should not be considered as formal
requirement. For example, some technologies may violate these strict
layers and still be able to deliver a DetNet service.

NEW:
Distinguishing the function of two DetNet data plane layers, the
DetNet service sub-layer and the DetNet transport sub-layer, helps to
explore and evaluate various combinations of the data plane solutions
available, some are illustrated in Figure 4. This separation of
DetNet data plane layers, while helpful, should not be considered as
formal requirement. For example, some technologies may violate these
strict layers and still be able to deliver a DetNet service.

> * Page 9
>
> A DetNet node may have other resources requiring allocation and/or
> scheduling,
>
> This is just one of several examples for inconsistent use of terminology.
> What is a "DetNet node"? That term is not introduced in Section 2.1
DetNet node is an umbrella term when it is not needed to specify which 
type of DetNet node is mentioned.

A definition will be added:

DetNet node
    A DetNet intermediate node, a DetNet edge node, a DetNet relay node, 
or a DetNet transport node.


> * Page 14
>
> A DetNet network supports the dedication of a high proportion (e.g.
> 75%) of the network bandwidth to DetNet flows.
>
> The 75% value is not reasoned. What prevents using 99% of the 
> bandwidth for
> DetNet traffic?
The point is that even the majority of the flows can be DetNet flows not 
just a minority.

The example will be deleted:

OLD:
A DetNet network supports the dedication of a high proportion (e.g.
75%) of the network bandwidth to DetNet flows.

NEW:
A DetNet network supports the dedication of a high proportion of
  the network bandwidth to DetNet flows.


> * Page 15: Figure 2
>
> If the term "transport layer" cannot be avoided, the labels in this figure
> should at least be expanded to "DetNet transport layer".
Suggest to use "DetNet transport sub-layer" as described above.


> * Page 18: Figure 4
>
> As already mentioned earlier, Figure 4 is confusing. UDP is a transport
> protocol. If the term "transport" cannot be avoided, the labels in this
> figure should at least be expanded to "DetNet transport".
Suggest to use "DetNet transport sub-layer"as described above.


> * Page 23
>
> If the source transmits less data than this limit
> allows, the unused resource such as link bandwidth can be made
> available by the system to non-DetNet packets.
>
> Could there be additional requirements on the use of unused resources by
> non-DetNet packets, e.g., regarding preemption? I am just wondering... If
> that was possible, a statement like "... can be made available by the 
> system
> to non-DetNet packets as long as all guarantees are fulfilled" would be on
> the safe side, no?
The text will be updated as suggested, i.e., using:
"... can be made available by the system to non-DetNet packets as long 
as all guarantees are fulfilled"


> * Page 27:
>
> DetNet achieves congestion protection and bounded delivery latency by
> reserving bandwidth and buffer resources at every hop along the path
> of the DetNet flow.
>
> Why does this sentence use the word "hop"? As far as I understand, in 
> DetNet
> bandwidth and buffer resources are reserved in each DetNet 
> intermediate node.
> If there were hops over IP routers not being DetNet intermediate nodes, no
> resources would be reserved there. As per Section 4.3.3, it is possible to
> deploy DetNet this way. And obviously there can be resource 
> bottlenecks below
> IP, on devices that are not routers... So does "hop" here refer to IP 
> router
> hops or also to devices not processing IP (or IP/MPLS)?

IP/MPLS is meant as it is in scope of DetNet.

The sentence will be updated to

OLD:
DetNet achieves congestion protection and bounded delivery latency by
reserving bandwidth and buffer resources at every hop along the path
of the DetNet flow.

NEW:
DetNet achieves congestion protection and bounded delivery latency by
reserving bandwidth and buffer resources at each DetNet node along the
path of the DetNet flow.


> * Page 27:
>
> Standard queuing and transmission selection algorithms allow a
> central controller to compute the latency contribution of each
> transit node to the end-to-end latency, ...
>
> The text does not explain why a _central_ controller is needed for this
> computation. Why would a distributed control plane not be able to realize
> this computation. Isn't this implementation-specific?
It depends on the mechanism used whether or not a distributed control 
plane can be used. For instance, IEEE 802.1Qbv requires central control 
to compute the time-based schedule at each node.
As traffic engineering is needed, some entity has to perform the 
necessary computations, which is typically centralized. It can be a 
management entity if distributed control is applied.
Another aspect is that standard queuing algorithms are needed to be able 
to do it.

To make it clearer, the sentence will be updated to:

OLD:
Standard queuing and transmission selection algorithms allow a
central controller to compute the latency contribution of each
transit node to the end-to-end latency, to compute the amount of
buffer space required in each transit node for each incremental
DetNet flow, and most importantly, to translate from a flow
specification to a set of values for the managed objects that control
each relay or end system.

NEW:
Standard queuing and transmission selection algorithms allow
traffic engineering (Section 4.4.) to compute the latency contribution 
of each
DetNet node to the end-to-end latency, to compute the amount of
buffer space required in each DetNet node for each incremental
DetNet flow, and most importantly, to translate from a flow
specification to a set of values for the managed objects that control
each DetNet relay or end system.


> * Page 32
>
> To somebody who is not deeply familiar with DetNet, it is impossible 
> to parse
> the description of the examples in Section 4.7.3. For instance, "VID +
> multicast MAC address" is not introduced. I think this example must be
> expaned with additional context and explanation to be useful to readers.

"ETH-ID" is used as generic Ethernet ID, e.g., in the figure.

In order to avoid the use of terms not explained in detail, the text 
will be updated to

OLD:
End system "IP-A" uses the original App-flow specific ID ("L3-ID"),
but as it is connected to an Ethernet domain it has to push an
Ethernet-domain specific flow-ID ("VID + multicast MAC address",
referred as "ETH-ID") before sending the packet to "ETH-1" node.

NEW:
End system "IP-A" uses the original App-flow specific ID ("L3-ID"),
but as it is connected to an Ethernet domain it has to push an
Ethernet-domain specific flow-ID ("ETH-ID") before sending the
packet to "ETH-1" node.


> * Page 34
>
> There are three classes of information that a central controller or
> distributed control plane needs to know that can only be obtained
> from the end systems and/or nodes in the network.
>
> Wouldn't it be sufficient to state "Provisioning of DetNet requires 
> knowledge
> about ...". Does it matter in this context whether the provisioning is 
> done
> by a central controller or a distributed control plane? For instance, 
> could
> the same paragraph also apply to a network that uses _multiple_ central
> controllers, or hybrid combinations of central controllers and distributed
> control planes? In general, an architecture document should be agnostic to
> implementation aspects unless there is a specific need. In this specific
> case, I fail to see a need to discuss the realization of the control 
> plane of
> a network.
The text will be updated as suggested:

OLD:
There are three classes of information that a central controller or
distributed control plane needs to know that can only be obtained
from the end systems and/or nodes in the network. When using a peerto-
peer control plane, some of this information may be required by a
system’s neighbors in the network.
o Details of the system’s capabilities ...

NEW:
Provisioning of DetNet requires knowledge about:
o Details of the system’s capabilities ...


> Editorial nits:
>
> * Page 9:
>
> The low-level mechanisms described in Section 4.5 provide the
> necessary regulation of transmissions by an end system or
> intermediate node to provide congestion protection. The allocation
> of the bandwidth and buffers for a DetNet flow requires provisioning
> A DetNet node may have other resources requiring allocation and/or
> scheduling, that might otherwise be over-subscribed and trigger the
> rejection of a reservation.
>
> Probably a full stop is missing after "provisioning".
Full stop will be added.


> * Page 11: "... along separate (disjoint non-SRLG) paths ..."
>
> I find this confusing. I would understand e.g. "along separate
> (SRLG-disjoint) paths".
The text will be updated as suggested.


> * Page 34:
>
> When using a peer-
> to-peer control plane, some of this information may be required by a
> system's neighbors in the network.
>
> Would "acquired" be a better term?
This text goes away with the update to "Provisioning of DetNet requires 
knowledge about"


> * Page 34:
>
> o The identity of the system's neighbors, and the characteristics of
> the link(s) between the systems, including the length (in
> nanoseconds) of the link(s).
>
> "Latency" or "delay" would probably be a better terms if the value is
> measured in nanoseconds.
the text will be updated to:

OLD:
The identity of the system’s neighbors, and the characteristics of
the link(s) between the systems, including the length (in
nanoseconds) of the link(s).

NEW:
The identity of the system’s neighbors, and the characteristics of
the link(s) between the systems, including the latency of the link(s)
(in nanoseconds) .


> * Page 35:
>
> DetNet is provides a Quality of Service (QoS), and as such, does not
> directly raise any new privacy considerations.
>
> Broken sentence
"is" will be deleted


> * Please expand acronyms on first use (e.g., OTN)
>
acronyms will be resolved at first occurrence