]> Futurewei Technologies

US haoyu.song@futurewei.com

Southeast University

CN wy25@seu.edu.cn

Huawei Technologies

CN zhukeyi@huawei.com

Routing Fast Network Notifications FANN IOAM SRv6 congestion control load balancing This document describes FALCON (FAst Latency and COngestion Notification), a method that allows a traffic source to learn the queuing delay and congestion status of a path with a notification lag no greater than the one-way propagation delay from the congested node back to the source, which is at most about half of the baseline Round-Trip Time (RTT). FALCON combines in-network telemetry and source routing: a forward packet records the path it traverses, and the receiver returns a high-priority packet that is source-routed along the exact reverse of that path, collecting the state of the forward-direction queues as it passes. Where a hop-by-hop flow control mechanism is used, such as Priority-based Flow Control (PFC) or a backpressure mechanism for lossless WAN transport, the returned packet can also collect the buffer state that determines when each node throttles its upstream neighbor, allowing the source to act before throttling occurs and to distinguish the root of congestion from nodes that are only throttled by it. The fresher telemetry enables more effective traffic steering and congestion control. FALCON is applicable to data center and wide area networks within a single administrative domain and is designed to be realized with IETF In situ OAM (IOAM) and SRv6, with the extensions identified in this document.

Introduction Many congestion control (CC) and load balancing (LB) schemes rely on timely path congestion status and/or measurements of packet delay as the basis for rate adjustment or path selection. The motivation and requirements for fast network notifications are described in , and an overall architecture is described in . End-to-end feedback mechanisms, whether in-band (e.g., ECN echoed by the receiver, or in-network telemetry echoed in acknowledgments as in ) or out-of-band (e.g., mesh probing), deliver the state of a network node to the traffic source only after the carrying packet has completed the remainder of the forward path and the entire return path. The information is therefore at least one baseline RTT old for the first hop and, because RTT grows with queuing, becomes staler as the path becomes more congested, which is exactly when fresh information is most needed. For example, when a packet experiences congestion and is ECN-marked, the congestion itself delays the delivery of the mark to the source. In a Wide Area Network (WAN), where the baseline RTT can be tens to hundreds of milliseconds, a reaction based on such stale feedback may be ineffective or even counterproductive. The root cause of the staleness is that the state is sampled on the forward direction and must then travel the remaining forward path plus the full reverse path. FALCON instead samples the state on the reverse direction: the receiver returns a packet that is source-routed along the exact reverse of the forward path and that reads, at each node, the state of the queue that the forward traffic uses. The lag between sampling at a node and delivery to the source is then only the one-way delay from that node to the source. FALCON is designed to be realized with IOAM trace options and SRv6 . identifies the extensions that are needed to do so.

Terminology

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Definitions

INT:

In-Network Telemetry. A data packet or a dedicated probe packet carries an instruction header that causes network nodes on its path to add node data to the packet. The IOAM trace options are an instance of INT.

SR:

Source Routing. The originator of a packet specifies the path of the packet as an ordered list of segments. SRv6 realizes SR over IPv6.

Source (S):

The node that sends the monitored traffic and consumes FALCON notifications. S may be a host, a NIC or DPU, or an ingress router/switch that steers traffic.

Receiver (R):

The node that receives the forward packet and generates the FALCON Return Packet.

Transit Node (S_i):

The i-th FALCON-capable router or switch on the forward path from S to R, i = 1..n.

Forward Packet (P):

A packet from S to R, either a data packet of a monitored flow or a dedicated probe, that carries an INT instruction to record the path.

FALCON Return Packet (P'):

A packet generated by R in response to P, source-routed along the reverse of the path recorded by P, that collects the state of the forward-direction queues and delivers it to S.

Forward Queue of S_i:

The egress queue of S_i, for the traffic class of the monitored traffic, on the interface through which P left S_i. This is the same physical interface on which P' arrives at S_i.

Hop-by-Hop Flow Control (HFC):

Any mechanism by which a node, based on its own buffer state, signals its upstream neighbor to stop, slow down, or resume transmission of a class of traffic. The class (the "flow-control class") may be a priority, a slice, a tunnel, or a flow, depending on the mechanism. Examples include PFC , credit-based link-level flow control, and hop-by-hop backpressure for lossless transport over WAN .

Throttle Signal:

The HFC signal that stops or slows the upstream neighbor, e.g., a PFC PAUSE frame, a withheld or reduced credit grant, or a backpressure message.

Flow-Control Buffer of S_i:

The buffer accounting of S_i whose state determines when S_i sends a Throttle Signal to its upstream neighbor for the flow-control class of the monitored traffic, on the interface through which P entered S_i (e.g., the ingress priority group counter in a shared-buffer switch using PFC, or the per-slice buffer used by a WAN backpressure mechanism). This is the same physical interface through which P' leaves S_i.

Trigger Threshold:

The occupancy of the Flow-Control Buffer at which S_i sends a Throttle Signal, e.g., the XOFF threshold of PFC. It may be static or dynamic.

Baseline RTT:

The RTT between S and R excluding queuing delay, i.e., the sum of propagation, serialization, and minimum forwarding delays in both directions.

Staleness:

For a given piece of node state, the time between the instant it is sampled at the node and the instant it is available at S.

Problem Statement and Design Goals A source S needs to sense the current status of the path toward a destination (e.g., per-hop queuing delay, congestion, total path latency) in order to react in time. For state that originates at a node S_i, the staleness at S cannot be smaller than the one-way delay from S_i to S. For the most distant node on a path with symmetric propagation delays, this lower bound is half the baseline RTT. It is usually well below half of the measured RTT, which additionally includes queuing delay. FALCON aims to approach this lower bound. Its design goals are:

Accurate:

The notification reflects the state of the queues actually used by the monitored traffic on the path in question.

Timely:

The staleness of the state of each node approaches the one-way delay from that node to the source.

Lightweight:

The mechanism imposes low packet and bandwidth overhead, requires no packet generation at transit nodes, and needs no per-flow state in the network.

Reuse of existing standards:

The mechanism is built on existing IETF data-plane protocols, with the minimum extensions needed.

Solution Overview illustrates the operation.

FALCON Operation |-- P --->|---- ... ----->|-- P --->| | INT: record node ID and interfaces at S_i | | | | | | (2) | | | | R builds P': | | | | SRH = reverse | | | | of recorded path | | | | | (3) |<-- P' --|<-- P' --|<--- ... ------|<-- P' --| | high priority; at each S_i read the | | Forward Queue (egress queue of the | | interface on which P' arrived) and, with | | HFC, the Flow-Control Buffer of the | | interface through which P' leaves | | | | | | (4) S computes per-hop/path queuing delay and congestion ]]>

1. S sends a Forward Packet P (a data packet of flow F or a dedicated probe) carrying an INT instruction that causes each transit node S_i to record its node identifier and the identifiers of the interfaces through which P entered and left it.
2. On receiving P, R constructs a FALCON Return Packet P'. P' carries a segment list that steers it through S_n, ..., S_2, S_1 to S, pinned to the same links P used (), and an INT instruction to collect Forward Queue state.
3. P' is sent in a high-priority traffic class, so it experiences negligible queuing on the return path. When P' arrives at S_i on interface r, P left S_i through the same interface r, so S_i reports the state of the egress queue of r for the traffic class of the monitored traffic. Where HFC is deployed, S_i also reports the state of the Flow-Control Buffer, i.e., of the interface q through which P' leaves S_i and through which P entered it ().
4. S uses the collected data to derive per-hop and path queuing delay, the location and severity of congestion, and the total path delay (), and feeds these to its CC or LB function.

Because each S_i is sampled when P' passes it, the staleness of the state of S_i at S is the one-way delay from S_i to S. The state of the congested node that matters most to S is therefore delivered as quickly as any notification originating at that node could be, without requiring transit nodes to generate packets.

Detailed Operation

Forward Path Recording P carries an INT trace instruction requesting, at each FALCON transit node, a node identifier and the ingress and egress interface identifiers. With IOAM , this corresponds to the node_id and the ingress_if_id/egress_if_id data fields of the IOAM trace option. P also indicates the traffic class of the monitored traffic if it differs from the traffic class in which P itself is forwarded. If P is a data packet, S SHOULD apply the recording instruction only to a sample of packets (). If P is a dedicated probe, it MUST be forwarded along the same path as the monitored flow, e.g., by using the same ECMP-relevant header fields or the same SR segment list. If S already steers the monitored traffic with an explicit SR segment list whose segments identify every hop, recording the path is unnecessary; R can derive the return path from that segment list, or S can itself provide the return segment list in P.

Return Packet Construction R maps each recorded (node identifier, interface identifier) pair to an SRv6 SID. To guarantee that P' arrives at S_i on the interface through which P left S_i, the segment list SHOULD use, for each hop, an adjacency SID (e.g., End.X ) of S_(i+1) for the link between S_(i+1) and S_i, where S_(i+1) is identified by the recorded node identifier and the link by the recorded ingress interface of S_(i+1). Node SIDs alone are insufficient where parallel links or ECMP exist between adjacent nodes. The final segment identifies S. The mapping from IOAM node and interface identifiers to SIDs is provisioned by the operator or learned via the control plane; alternatively, where the deployment permits it, the transit nodes MAY record the relevant SID directly. The mechanism for distributing this mapping is outside the scope of this version of the document. The length of the segment list grows with the hop count. To bound the overhead, particularly in the WAN, compressed SIDs SHOULD be used where supported. P' MUST be marked with a DSCP that maps to a high-priority, low-latency per-hop behavior within the FALCON domain, and it carries an INT instruction requesting the Forward Queue data of , together with the monitored traffic class. P' MAY be piggybacked on a transport-layer acknowledgment or congestion notification packet that R would send to S anyway.

Per-Hop Processing on the Return Path When P' arrives at S_i on interface r, S_i reports the state of the Forward Queue, i.e., the egress queue of interface r for the monitored traffic class. Depending on the capability of S_i and the requested data, S_i reports one or more of:

• the Forward Queue depth, in bytes;
• an estimate of the Forward Queue queuing delay, preferably derived from hardware sojourn-time measurement or the recent dequeue rate of the queue;
• a congestion indication, set when the Forward Queue depth or delay exceeds a configured threshold (e.g., the ECN marking threshold of that queue);
• the transmit utilization of interface r.

Note that the existing IOAM queue depth data field reports the egress queue of the interface through which the IOAM packet itself leaves the node, which is not the Forward Queue. A new data field is therefore needed (). The congestion indication is carried as telemetry data in P'. It MUST NOT be signaled by setting the ECN field of the IP header of P', since that field reflects congestion experienced by P' itself.

Hop-by-Hop Flow Control State In lossless networks, a node S_i protects its buffer by sending a Throttle Signal to its upstream neighbor S_(i-1) when the occupancy of its Flow-Control Buffer exceeds the Trigger Threshold, and by releasing it when the occupancy falls sufficiently. Examples are PFC in RoCEv2 data center fabrics, credit-based link-level flow control, and hop-by-hop backpressure applied per slice or per flow over a WAN . The mechanisms differ in the signal they use and in the granularity of the flow-control class, but share the same structure: a per-class buffer accounting on the receiving side of a link, a threshold, and a signal that throttles the sending side. The Flow-Control Buffer is accounted separately from the Forward Queue, and its Trigger Threshold may be dynamic. The depth of the Forward Queue therefore does not by itself indicate how close S_i is to throttling its upstream neighbor. When P' leaves S_i through interface q, S_i MAY report, for the flow-control class of the monitored traffic, one or more of:

• the Flow-Control Buffer occupancy, in bytes;
• the headroom remaining before the Trigger Threshold currently in effect, or the occupancy as a fraction of that threshold, which is comparable across nodes with different buffer sizes and threshold settings;
• an indication that S_i is currently throttling its upstream neighbor on interface q;
• an indication that the Forward Queue is currently throttled by S_(i+1) on interface r, optionally with the throttled duration or the number of Throttle Signals received in a recent interval.

The reported information is expressed independently of the specific HFC mechanism, so that the source can interpret it uniformly across a path that may combine, for example, PFC within data centers and a different backpressure mechanism on the interconnecting WAN. These data are sampled when P' passes S_i and are therefore subject to the same staleness bound as the Forward Queue state (). With them, S can:

• Act before throttling occurs. A shrinking headroom is an early warning that lets S reduce its rate or move traffic before a Throttle Signal is sent, reducing the frequency and duration of throttling and its side effects: head-of-line blocking, congestion spreading to unrelated traffic, throttle storms, and deadlock.
• Locate the root of congestion. Throttling propagates upstream, so several nodes may show large queues. A node whose Forward Queue is large but not throttled is a congestion root; upstream nodes whose Forward Queues are throttled are victims of backpressure. S can then react to the root, e.g., by avoiding the root link in load balancing, instead of to the victims.
• Estimate queuing delay correctly. While a Forward Queue is stopped or slowed by a Throttle Signal, its service rate is reduced or zero, so dividing its depth by the link rate underestimates its delay ().

HFC reacts hop by hop, within the headroom of a single link. FALCON does not replace it: a burst shorter than the control loop between S and S_i can still trigger throttling. The purpose of this data is to let S reduce sustained buildup that would otherwise lead to repeated or prolonged throttling. This benefit is greatest where the headroom per link is costly relative to the link delay, as on long-distance WAN links. In some devices, flow-control buffer accounting and throttle state are kept in the buffer manager and are not directly accessible while a packet of a different interface is processed. A node that cannot provide a requested value SHOULD mark it as unavailable rather than omit it.

Per-Hop Collection versus On-Path Aggregation If transit nodes can perform simple arithmetic in the data plane, P' MAY carry aggregate values instead of a per-hop list: the running sum of Forward Queue delay (yielding the path queuing delay), or the maximum Forward Queue depth or delay together with the identifier of the node reporting it (yielding the bottleneck), updated by a compare-and-replace operation. Where HFC state is collected, P' MAY similarly carry the minimum headroom to the Trigger Threshold along the path with the identifier of the node reporting it, and a flag set by any node that is throttling or being throttled. Aggregation keeps the size of P' constant regardless of path length. If transit nodes cannot aggregate, P' collects per-hop values, and S performs the computation using its knowledge of link capacities.

Processing at the Source The delay of P over the path consists of a baseline component (propagation, serialization, and minimum forwarding delay) and a queuing component. The queuing component is the sum of the Forward Queue delays reported by P'. Where a node reports only queue depth, S estimates the queuing delay at S_i as the queue depth divided by the service rate of that queue. The service rate equals the link rate only when the monitored class is the sole active class on the link and is not subject to flow control (e.g., HFC throttling, see ) or shaping; S SHOULD prefer node-reported delay estimates when available. The baseline component can be obtained by the source as the minimum observed RTT (or minimum P-to-P' round trip) over a suitable window, divided between directions under an assumption of symmetric propagation delay; or, where clocks are synchronized, from IOAM timestamps carried in P and P'. The estimated current path delay of P is the baseline component plus the collected queuing component. S passes the per-hop state, path queuing delay, bottleneck information, and congestion indications to its CC or LB function. How S reacts is outside the scope of this document.

Freshness Analysis Let d(X,Y) denote the one-way delay from X to Y without queuing, and Q_fwd(i) and Q_rev the queuing delays encountered after S_i on the forward path and on the reverse path from R to S, respectively. For the state of S_i:

• With forward-direction collection that is echoed back by R, the staleness is d(S_i,R) + Q_fwd(i) + d(R,S) + Q_rev. Even without queuing, this is at least d(S_i,R) + d(R,S).
• With FALCON, P' traverses the return path in a high-priority class, so the staleness is approximately d(S_i,S), which is at most d(R,S), or about half the baseline RTT.

The gain is largest for nodes close to the source and grows with the amount of forward-path queuing. Since a reaction by S takes effect at S_i only after d(S,S_i), the FALCON control loop for a bottleneck at S_i is approximately the baseline RTT between S and S_i, which is the physical lower bound for any source-based reaction. Staleness is distinct from sampling frequency: the interval between consecutive P' packets is determined by the sampling rate of P (). In addition, the per-hop samples carried by one P' are taken at slightly different times (S_n first, S_1 last), spread over d(S_n,S_1).

Implementation Considerations and Gap Analysis

Encapsulation In IPv6 networks, P can carry the IOAM trace option in an IPv6 Hop-by-Hop Options header . For P', which carries both an SRH and an IOAM trace, two approaches are possible:

• IOAM in a Hop-by-Hop Options header together with the SRH ; or
• IOAM data carried in a UDP payload with an SRH flag indicating segment-by-segment processing, as proposed in . Since the segment list of P' contains a segment for every hop, each transit node is a segment endpoint and processes the IOAM data.

Gaps The following extensions are needed for a standards-based realization:

1. An IOAM data field reporting the state (depth, and optionally delay and congestion indication) of the egress queue of the interface on which the IOAM packet was received, for a specified traffic class.
2. An IOAM data field, independent of the specific HFC mechanism, reporting for a specified flow-control class the Flow-Control Buffer state (occupancy, and optionally headroom to the Trigger Threshold and a throttling-upstream indication) of the interface through which the IOAM packet leaves the node, and a throttled-by-downstream indication for the egress queue of the interface on which it was received. The existing IOAM queue depth field reports egress queue occupancy, not flow-control buffer accounting.
3. IOAM data fields or options for on-path aggregation (sum, maximum or minimum with location, and flags) as described in .
4. A means for P and P' to indicate the monitored traffic class when it differs from the packet's own class.
5. A standard encapsulation of IOAM in SRv6 packets, e.g., .
6. An indication in P that requests R to generate P', and the parameters for P' (requested data, return traffic class).
7. A mechanism to map IOAM node and interface identifiers to SRv6 adjacency SIDs, or to record SIDs directly.

Operational Considerations

Sampling Rate and Overhead Generating a P' for every packet is unnecessary and costly at high link rates. S SHOULD request recording on a configurable subset of packets, e.g., once per configured time interval, per configured number of bytes, or per RTT per flow. R MUST rate-limit the generation of P', and the aggregate volume of P' in the high-priority class MUST be bounded so that it cannot starve other traffic in that class. The trace data added to P can cause P to exceed the path MTU. S SHOULD account for the maximum trace size when selecting data packets to carry the instruction, or use dedicated probes. The HFC data of are useful only where an HFC mechanism is enabled for the monitored traffic; S SHOULD NOT request them otherwise.

Partial Deployment Nodes that do not support FALCON do not record themselves in P and are therefore not included in the return segment list; the link state between two FALCON nodes separated by a non-capable node cannot be observed, and P' traverses the intervening segment using normal forwarding, which may not follow the reverse of the forward path. S SHOULD detect gaps (e.g., via hop-count or TTL information in the trace) and treat the corresponding portion of the path as unobserved.

Path Changes, Link Aggregation, and Asymmetry P' reflects the path taken by the specific P from which it was generated. If the monitored flow is re-hashed or re-routed, S SHOULD discard state for the old path. Where a link is a member of a link aggregation group, the Forward Queue is per member link; the recorded interface identifier SHOULD identify the member link. FALCON requires only that each link be bidirectional; it does not require symmetric routing, because P' is explicitly steered. Estimates of the baseline delay that assume symmetric propagation are affected by asymmetric link delays.

Applicability FALCON is intended for use within a single administrative domain that is both an SR domain and an IOAM domain, consistent with the deployment scope of . In a data center network the paths are short and the main benefit is sub-RTT reaction to incast and transient congestion. In a WAN the baseline RTT is large, and the absolute reduction in staleness is correspondingly large.

Security Considerations The security considerations of IOAM and SRv6 apply. In particular, FALCON packets MUST be confined to a domain that is both an SR domain and an IOAM domain; packets carrying FALCON instructions, SRHs, or the P' traffic class MUST be filtered at the domain boundary. Additional threats specific to FALCON include:

• Forged or modified P' packets could induce S to reduce its rate unnecessarily or to move traffic onto a worse path. Integrity protection of IOAM data can mitigate modification in transit; S SHOULD accept P' only if it corresponds to an outstanding P (e.g., by matching a sequence number or nonce carried in P).
• R constructs a segment list from data recorded in P. A crafted P could cause R to emit P' along arbitrary paths or toward a victim. R MUST verify that all SIDs it places in P' belong to the local domain and that the final destination is the source address of P, and MUST rate-limit P' generation.
• P' uses a high-priority class. If that class is not protected, it could be abused to obtain preferential treatment or to starve other high-priority traffic. The class SHOULD be reserved and policed.
• P' reveals topology, queue occupancy and, where collected, buffer thresholds and HFC state to S. Where S is an end host that is not fully trusted by the operator, the operator SHOULD consider limiting the data returned, e.g., to aggregate values only.

IANA Considerations This document has no IANA actions at this time. Future versions are expected to request allocations for the extensions listed in , e.g., new bits in the IOAM Trace-Type registry and any flags or code points required for the P' request indication.

References Normative References Informative References IEEE et al. Proceedings of ACM SIGCOMM 2019 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)

Comparison with Related Mechanisms

Receiver-echoed telemetry (e.g., ECN echo, ):

State is sampled on the forward path and returned in acknowledgments. Staleness is at least one baseline RTT for the first hop and includes forward and reverse queuing.

Switch-originated notifications (e.g., , IEEE 802.1Qau QCN, FANN push-based notifications):

The congested node generates a notification directly toward the source. Staleness is the same lower bound as FALCON, d(S_i,S), but transit nodes must generate packets and must know or derive the source to notify. FALCON requires only in-band data insertion at transit nodes, and the returned state is scoped to the queues used by the source's own traffic, covering the whole path in one packet.

IOAM Loopback and IOAM Direct Export :

Loopback causes transit nodes to return a copy of the packet to the encapsulating node, and Direct Export sends data to a collector. Both carry state sampled when the forward packet passed the node, and neither reads the forward queue on the return path.