<?xml version="1.0" encoding="utf-8"?>
<!-- name="GENERATOR" content="github.com/mmarkdown/mmark Mmark Markdown Processor - mmark.miek.nl" -->
<rfc version="3" ipr="trust200902" docName="draft-heist-tsvwg-ecn-deployment-observations-02" submissionType="IETF" category="info" xml:lang="en" xmlns:xi="http://www.w3.org/2001/XInclude" consensus="true">

<front>
<title abbrev="ecndep">Explicit Congestion Notification (ECN) Deployment Observations</title><seriesInfo value="draft-heist-tsvwg-ecn-deployment-observations-02" stream="IETF" status="informational" name="Internet-Draft"></seriesInfo>
<author initials="P." surname="Heist" fullname="Peter G. Heist"><organization></organization><address><postal><street></street>
<city>Liberec 30</city>
<code>463 11</code>
<country>Czech Republic</country>
</postal><email>pete@heistp.net</email>
</address></author>
<author initials="J." surname="Morton" fullname="Jonathan Morton"><organization></organization><address><postal><street>Kokkonranta 21</street>
<city>Pitkajarvi</city>
<code>31520</code>
<country>Finland</country>
</postal><phone>+358 44 927 2377</phone>
<email>chromatix99@gmail.com</email>
</address></author>
<date/>
<area>Internet</area>
<workgroup>Transport Working Group</workgroup>
<keyword>Internet-Draft</keyword>
<keyword>ECN</keyword>
<keyword>congestion control</keyword>

<abstract>
<t>This note presents data gathered at an Internet Service Provider's gateway on
the observed deployment and usage of ECN. Relevant IP counter and flow tracking
data was collected and analyzed for TCP and other protocols.</t>
</abstract>

</front>

<middle>

<section anchor="introduction"><name>Introduction</name>
<t>To help guide the evolution of ECN, there is a need for more data on current
deployment status, and observed usage of the ECN related bits, including:</t>

<ul>
<li>the initiation and acceptance of ECN capable TCP flows</li>
<li>marking via CE, and feedback for TCP via the ECE and CWR flags</li>
<li>codepoints set on packets for protocols other than TCP</li>
</ul>
<t>For several weeks, we gathered data on all traffic through an Internet Service
Provider's gateway. Though some of the results are informative, we caution that
a larger, more widely reviewed and geographically distributed survey would be
needed to be authoritative.</t>
</section>

<section anchor="collection-details"><name>Collection Details</name>
<t>From December 28, 2020 to January 20, 2021, data was gathered on all traffic
into and out of the Internet gateway at FreeNet Liberec, a cooperative WISP in
an urban area of the Czech Republic. A total of 122.5 TB of incoming data and 12
TB of outgoing data was seen.</t>
<t>Around 660 members belong to the ISP, and 861 member IP addresses on the LAN
were considered active during data collection. Most member IPs are used by a
household of users, while others are for individual devices and public
locations.</t>
<t><xref target="IPTABLES-ECN"></xref> was used to collect and analyze the data. This consists of a
script to gather the data using iptables and ipsets in Linux, and an analysis
program that produces textual output. An abbreviated version of the output is
included in <xref target="ecn-stats-output"></xref>. See the referred to source repository for more
details and full output.</t>
</section>

<section anchor="observations"><name>Observations</name>
<t>Our key observations are summarized as follows, and further expanded upon in the
following sections:</t>

<ul>
<li>1.44% of TCP flows attempted to initiate ECN, across 390 member IPs (45%).</li>
<li>The acceptance rate for ECN flows was likely &gt;50%.</li>
<li>382 member IPs (44%) successfully negotiated any TCP ECN flows, determined
by receipt of any ECN SYN/ACKs.</li>
<li>71 member IPs (18.6% of ECN negotiating) saw possible AQM marking via CE and
ECE, with 38 in one of two subnets with known AQMs, and 33 from other subnets,
with unknown, possible AQMs.</li>
<li>The 33 IPs that saw possible AQM marking from unknown, possible AQMs, were
from a population of 319 ECN negotiating IPs not in subnets with known AQMs,
leading to a rough estimate of 10.3% of random paths with AQMs present.</li>
<li>Nonzero ECN codepoints were observed on 0.053% of non-TCP packets, with
possible attribution to tunneled ECN and/or misuse of the ECN field.</li>
</ul>

<section anchor="ecn-endpoint-activity"><name>ECN Endpoint Activity</name>

<section anchor="client-initiation"><name>Client Initiation</name>
<t>Of 319.5 million TCP SYNs from LAN to WAN, 1.44% indicated ECN capability. Of
861 active member IP addresses, 390 (45.3%) attempted initiation for at least
one ECN flow. A large proportion of the ECN flows are thought to come from Apple
devices.</t>
</section>

<section anchor="server-acceptance"><name>Server Acceptance</name>
<t>While 4.6 million ECN TCP SYNs were seen from LAN to WAN, 3.3 million ECN
SYN-ACKs were seen in return. While it's not possible to get an exact ECN
acceptance rate from this, it appears to be reasonably high, likely due to
default acceptance on prevailing server operating systems like Linux, FreeBSD
and recent versions of Windows Server.</t>
</section>
</section>

<section anchor="rfc3168-aqm-activity"><name>RFC3168 AQM Activity</name>
<t>There appears to be some evidence of <xref target="RFC3168"></xref> marking AQMs. Of 861 active
member IP addresses:</t>

<ul>
<li><t>382 member IPs, or 44%, successfully negotiated any TCP ECN flows</t>

<ul>
<li>63 of those (16.5%) were from subnets with known AQMs deployed</li>
<li>319 (83.5%) were from subnets without a known AQM</li>
</ul></li>
<li>90 member IPs saw any CE or ECE marks on negotiated TCP ECN flows</li>
<li><t>71 member IPs (18.6% of ECN negotiating) saw possible AQM activity, of which:</t>

<ul>
<li>38 are in subnets with known AQMs (60.3% of 63 in total, representing the
detection rate)</li>
<li>33 are in subnets with unknown AQMs (10.3% of 319 in total, representing the
proportion of possible AQMs on random paths)</li>
</ul></li>
</ul>
<t>Some factors may affect the 10.3% proportion of possible AQMs on random
paths:</t>

<ul>
<li>There may be false positives and negatives in possible AQM identification.</li>
<li>We have not attempted to correct for the likelihood of missed AQMs, for
example by assuming a detection rate similar to that for the known AQMs.</li>
</ul>
<t>The following subsections clarify what is known AQM activity, what is considered
possible activity, and some of the limitations in identifying AQMs.</t>

<section anchor="known-aqm-activity"><name>Known AQM Activity</name>
<t>Two backhaul links have fq_codel <xref target="RFC8290"></xref> deployed, serving the 10.45.64.0/24
and 10.45.235.0/24 subnets (also see <xref target="affiliation"></xref>). These are the known AQMs
that account for the 38/71 member IPs that saw possible AQM activity. It is
possible that additional AQMs exist within the known subnets with AQM
deployments, however those would be harder to identify. The source of the AQM
activity from the remaining subnets is unknown, and could be from CPE, an
external network, or both.</t>
</section>

<section anchor="identifying-aqm-activity"><name>Identifying Possible AQM Activity</name>
<t>For detection of <xref target="RFC3168"></xref> AQM activity, we observe TCP flows. The ECE flag
seen on negotiated TCP ECN flows allows for a more accurate confirmation of
marking, since CE marks applied after traversal through the gateway will be
missed.</t>
<t>To classify a member IP as having seen AQM activity, we use the following
criteria:</t>

<ol>
<li>Nonzero ECT(0) counts in both directions</li>
<li>AND nonzero ECE in either direction</li>
<li>AND for nonzero CE and ECE, ECE:CE ratio &gt;= 2:1, OR subtraction of the ECE
marks from the ECE marks in the opposite direction allows the opposite
direction to meet the same criteria, as explained in the following section.</li>
</ol>
</section>

<section anchor="aqm-false-positives"><name>AQM False Positives</name>
<t>Some of the member IP addresses showed small and nearly identical counts of CE
marks in one direction and ECE flags in both directions, which seems unlikely to
be the result of genuine congestion and AQM activity. Our hypothesis is that
these are part of a &quot;fingerprinting&quot; routine used by port scanners when they
find an open port.</t>
<t>To exclude these, we looked for cases where inbound CE mark counts were within a
2x factor of outbound ECE flags, and discounted these as evidence of inbound AQM
activity. In these cases we also applied a more stringent check for outbound
AQM activity, by subtracting the outbound ECE flags from the inbound ECE flags,
and looking for at least a 2x ratio between any remainder and the number of
any outbound CE marks observed.</t>
</section>

<section anchor="limitations-in-aqm-identification"><name>Limitations in AQM Identification</name>
<t>It is possible that some false-positives and/or false-negatives still remain in
our classification method. Additionally, there is no way to know if we have
identified all utilized paths with AQMs present. Detection first requires an ECN
flow (1.44% of TCP SYNs requested ECN) and for that flow to encounter congestion
at a bottleneck with an AQM enabled.</t>
</section>
</section>

<section anchor="ecn-codepoints-non-tcp"><name>ECN Codepoints on Non-TCP Protocols</name>
<t>Referring to the packet counts in the <em>All IP</em> / <em>Both Directions</em> table in the
stats output in <xref target="ecn-stats-output"></xref>, where M indicates megapackets and G,
gigapackets:</t>
<table>
<thead>
<tr>
<th></th>
<th>TCP (X)</th>
<th>Conntrack (XX)</th>
<th>Other</th>
<th>Total</th>
</tr>
</thead>

<tbody>
<tr>
<td>All</td>
<td>76.60 G</td>
<td>-&gt;</td>
<td>43.52 G</td>
<td>120.14 G</td>
</tr>

<tr>
<td>CE</td>
<td>10031</td>
<td>3.38 M</td>
<td>813951</td>
<td>4.20 M</td>
</tr>

<tr>
<td>ECT(0)</td>
<td>523.91 M</td>
<td>9.66 M</td>
<td>2.55 M</td>
<td>536.12 M</td>
</tr>

<tr>
<td>ECT(1)</td>
<td>63</td>
<td>6.68 M</td>
<td>182928</td>
<td>6.86 M</td>
</tr>
</tbody>
</table><t>(X)  TCP ECN packet counts only for negotiated ECN flows</t>
<t>(XX) UDP, ICMP, DCCP, SCTP, GRE (Conntrack All packets included in Other)</t>
<t>We note the following, where TCP-ECN refers to TCP flows that have successfully
negotiated ECN support:</t>

<ul>
<li>TCP-ECN accounted for 97.7% of the 536 million ECT(0) marks</li>
<li>0.68% of all TCP packets were from TCP-ECN marked with a nonzero ECN codepoint</li>
<li>About 0.05% of all non-TCP packets were marked with a nonzero ECN codepoint</li>
<li>Not-TCP-ECN accounted for 99.8% of the 4.2 million CE marks</li>
<li>Not-TCP-ECN accounted for virtually all of the ECT(1) marks</li>
<li>6.4 of 6.68 million ECT(1) marks were to a single member IP</li>
</ul>
<t>Possible explanations for ECN marks on non-TCP packets are explored further in
this section.</t>

<section anchor="tunneled-ecn-traffic"><name>Tunneled ECN Traffic</name>
<t>There are several different encapsulation methods used when handling the ECN
field through tunnels, as per <xref target="RFC3168"></xref> and <xref target="RFC6040"></xref>:</t>

<ol>
<li>copy the ECN field from the inner to the outer packet</li>
<li>reset the ECN field on the outer packet to ECT(0)</li>
<li>set Not-ECT on the outer packet</li>
</ol>
<t>When method 3 is used at both ends of a tunnel, we would not expect to see
ECN codepoint usage in either direction.</t>
<t>When methods 1 or 2 are used at both ends of a tunnel, we would expect to see
ECT(0) on both incoming and outgoing packets. We would also expect a bias
towards incoming packets, since more data is generally downloaded than uploaded,
and pure ACKs do not have ECT(0) marks.</t>
<t>When method 3 is used at only one end of the tunnel, we would expect to see
ECT(0) on packets in only one direction.</t>
<t>We note the following:</t>

<ul>
<li>Bi-directional ECT(0) marks were observed for two member IP / port pairs,
on UDP port 443 and 60001.</li>
<li><t>Uni-directional ECT(0) marks were observed for:</t>

<ul>
<li>UDP port 4500 (IPSec NAT traversal <xref target="RFC3948"></xref>) with 23 member IP addresses
downstream, and 1 member IP address upstream.</li>
<li>UDP port 51820 <xref target="WIREGUARD"></xref> with 2 member IP addresses downstream.</li>
<li>Numerous UDP ports in other ranges, mostly on the downstream.</li>
</ul></li>
</ul>
<t>While it's possible that some of the data observed was from tunneled ECN
traffic, this can't be established definitively.</t>
</section>

<section anchor="quic-ecn"><name>QUIC-ECN</name>
<t>Since a production implementation of QUIC-ECN is known to exist, we could see
some of that traffic in the non-TCP data. One member IP / dstport pair to
udp:443 saw 4603 ECT(0) marks from the WAN, and 1882 ECT(0) marks from the LAN.
This may have been QUIC-ECN, although it can't be established definitively.</t>
</section>

<section anchor="use-of-the-ecn-field-for-historical-reasons"><name>Use of the ECN Field for Historical Reasons</name>
<t>Some applications may still use historical definitions of the former TOS byte.
Although RFC791 reserved the ECN field for future use, the now obsolete
<xref target="RFC1349"></xref> defined the TOS field as four bits within the Type of Service octet,
one of which overlaps with the ECN field. This may account for some of the
observed usage of ECT(0), since the value for &quot;minimize monetary cost&quot; was 0001,
shifted to the left one bit, coinciding with ECT(0).</t>
</section>

<section anchor="use-of-the-ecn-field-inadvertently"><name>Use of the ECN Field Inadvertently</name>
<t>Users of operating system's socket APIs wishing to set a DiffServ codepoint may
be confused as to whether or not they need to shift the desired value left two
bits before passing it in. Additionally, OS header files have been seen with
out-of-date definitions for obsolete values in the former Type of Service octet,
and obsolete definitions from <xref target="RFC2481"></xref>.</t>
<t>Another possible source of confusion is the TOS field values listed in the now
obsolete <xref target="RFC1349"></xref>, without having been shifted. A casual reader could see the
value 0001 for &quot;minimize monetary cost&quot; and think that they should use this
value in the TOS byte, conflicting with ECT(1), not realizing that:</t>

<ul>
<li><xref target="RFC1349"></xref> is obsolete</li>
<li>even if it weren't obsolete, the TOS values must be shifted to the left <strong>by
one bit</strong></li>
</ul>
<t>To reduce incorrect usages of the DS field, OS header files should be sanitized,
obsolete RFCs more prominently marked as such, and API documentation brought up
to date.</t>
</section>

<section anchor="use-of-the-ecn-field-maliciously"><name>Use of the ECN Field Maliciously</name>
<t>It's possible that some software is using the ECN field to gain an advantage in
Internet queues or for some other nefarious purpose. Further analysis would be
needed to determine if this is the case.</t>
</section>
</section>
</section>

<section anchor="study-limitations-and-recommendations-for-future-work"><name>Study Limitations and Recommendations for Future Work</name>
<t>The main limitation of this study is that we use counter data at different
levels of granularity. In some cases, it would be possible to gain more
certainty by taking packet captures and analyzing individual flows, but this can
be more difficult to do on production data, both for technical and privacy
reasons.</t>

<section anchor="ecn-acceptance-rate"><name>ECN Acceptance Rate</name>
<t>While we captured the ratio of ECN SYNs to ECN SYN-ACKs, we do not have an exact
count of flows that were accepted or rejected. It may be possible to do this
more accurately with additional iptables rules in <xref target="IPTABLES-ECN"></xref>. Additionally,
flows are tracked as ECN capable upon receipt of an ECN SYN/ACK, regardless of
whether that SYN/ACK was associated with an outgoing ECN SYN. This could be
improved with more state tracking.</t>
</section>

<section anchor="not-ect-counts-by-ip"><name>Not-ECT Counts by IP</name>
<t>While we captured the nonzero ECN codepoints by IP address, separately for TCP
and conntrack-supported protocols, we do not have a count of Not-ECT by IP
address. Although it may be possible to obtain this, it would require an ipset
lookup for every packet traversing the gateway.</t>
</section>

<section anchor="ecn-marked-packets-on-not-ecn-capable-tcp-flows"><name>ECN Marked Packets on Not-ECN-Capable TCP Flows</name>
<t>While we captured counts of ECN marked packets on ECN capable TCP flows,
identified upon receipt of an ECN SYN/ACK, we do not have separate counts of TCP
packets that were marked with ECN codepoints without having negotiated ECN. This
should be possible to obtain with additional iptables rules in <xref target="IPTABLES-ECN"></xref>.</t>
</section>

<section anchor="tunnels"><name>Tunnels</name>
<t>Tunnel protocols are challenging because of the different encapsulation methods
and protocols used. An analysis at the flow level, rather than by IP address and
destination port pairs, might be more useful in identifying the usage of ECN over
tunnels.</t>
</section>

<section anchor="non-tcp-protocols"><name>Non-TCP Protocols</name>
<t>More research is needed into the reasons for ECN codepoints being set on non-TCP
traffic. Given the relatively low volume of this traffic, it might be practical
to take packet captures of it for further analysis.</t>
<t>Additionally, we are currently not able to differentiate between the total
number of packets for conntrack-supported and Other protocols. This could be
improved with some changes to <xref target="IPTABLES-ECN"></xref>.</t>
</section>

<section anchor="other-protocols"><name>Other Protocols</name>
<t>While this study looked at signals by IP address for TCP and IP/port for
conntrack-supported protocols, it does not break down signals for Other
protocols by IP address. Among those protocols is IPSec ESP packets, using IP
protocol 50. The <xref target="IPTABLES-ECN"></xref> script could be modified to create more ipsets
of type hash:ip, parallel to what was done for IP traffic as a whole, to further
analyze these protocols for tunnel activity.</t>
</section>

<section anchor="cwr-flag"><name>CWR Flag</name>
<t>Counting occurrences of TCP's CWR flag could better help distinguish between
some of the false positives and negatives in AQM detection.</t>
</section>

<section anchor="ns-flag"><name>NS Flag</name>
<t>Since <xref target="RFC8311"></xref> declared that the NS (Nonce Sum) flag is again Reserved, after
its now historical use by <xref target="RFC3540"></xref>, we could collect any observed usages of
this flag, to confirm that it's available for use in practice.</t>
</section>
</section>

<section anchor="ecn-stats-output"><name>Abbreviated Output from ecn-stats</name>
<t>This abbreviated output only includes LAN to WAN flows, and a small subset of
the non-TCP conntrack protocols by member IP address. For full output, see the
<xref target="IPTABLES-ECN"></xref> repository.</t>
<t><strong>Note</strong> the IP addresses shown here have been anonymized within the 10.0.0.0/8
address space, in a way that retains the subnet structure.</t>

<section anchor="all-ip"><name>All IP</name>

<artwork>Packets, CE, ECT(0) and ECT(1) are packet counts, and use
units of M, G or T for mega, giga, or terapackets.

Total (both directions):

               TCP [*]     Conntrack [+]   Other      Total
               -------     -------------   -----      -----
    Bytes      101.22 TB   -&gt;              33.22 TB   134.46 TB
    Packets    76.60 G     -&gt;              43.52 G    120.14 G
    |-CE       10031       3.38 M          813951     4.20 M
    |-ECT(0)   523.91 M    9.66 M          2.55 M     536.12 M
    |-ECT(1)   63          6.68 M          182928     6.86 M

WAN to LAN:

               TCP [*]    Conntrack [+]   Other      Total
               -------    -------------   -----      -----
    Bytes      95.79 TB   -&gt;              26.65 TB   122.45 TB
    Packets    41.43 G    -&gt;              30.29 G    71.72 G
    |-CE       9298       3.38 M          721002     4.11 M
    |-ECT(0)   480.35 M   9.62 M          1.93 M     491.91 M
    |-ECT(1)   62         6.68 M          65111      6.74 M

LAN to WAN:

               TCP [*]   Conntrack [+]   Other     Total
               -------   -------------   -----     -----
    Bytes      5.43 TB   -&gt;              6.57 TB   12.00 TB
    Packets    35.17 G   -&gt;              13.23 G   48.41 G
    |-CE       733       60              92949     93742
    |-ECT(0)   43.56 M   40366           614623    44.21 M
    |-ECT(1)   1         28              117817    117846

    [*] TCP ECN packet counts only for negotiated ECN flows
        Counts for non-ECN TCP flows in Other
    [+] Conntrack protocols: UDP, ICMP, DCCP, SCTP, GRE
        Conntrack total Bytes and Packets included in Other
</artwork>
</section>

<section anchor="tcp-initiated-from-lan-to-wan"><name>TCP initiated from LAN to WAN</name>

<artwork>SYN packet count totals for active IPs:

    All SYNs:     319560652
    ECN SYNs:     4601118 (1.44% of all)
    ECN SYN/ACKs: 3273815 (71.15% of ECN SYNs)

ECN packet count totals for active IPs:

    Direction CE   ECE    ECT(0)    ECT(1)
    --------- --   ---    ------    ------
    From LAN  733  502985 42903861  1
    From WAN  9298 19367  479756419 62

IP address counts with TCP and ECN activity:

    Active (sent &gt;= 10 SYNs):  861 (of 1195)
    Initiated any ECN flows:   390 (45.3%)
    Negotiated any ECN flows:  382 (44.4%)
    Saw CE or ECE on ECN flow: 90 (23.6% of ECN negotiating)
    Saw ECT(1) on ECN flow:    5

IP address counts with possible AQM activity:

    Negotiated any ECN flows:       382
    |- in subnet with known AQM:    63 (16.5% of ECN negotiating)
    |- in subnet without known AQM: 319 (83.5% of ECN negotiating)

    Criteria for possible AQM activity:
        nonzero ECT(0) in both directions
        AND nonzero ECE in either direction
        AND ECE:CE ratio &gt;= 2:1 OR opposite direction
            ECE difference meets same criteria

    IPs with possible AQM activity: 71 (18.6% of ECN negotiating)
    |- from known AQMs:             38 (60.3% of 63 /w known AQM)
    |- from unknown, possible AQMs: 33 (10.3% of 319 w/o known AQM)

ECN flow packet counts by active IP, for nonzero CE or ECE:

    Flags column:
        A: possible AQM activity (see Criteria above)
        K: known AQM deployment

                        ECT(0)   CE   ECE    ECT(0)  CE   ECE
                        from     from from   from    from from
    IP            Flags WAN      WAN  LAN    LAN     LAN  WAN
    --            ----- ---      ---  ---    ---     ---  ---
    10.45.9.88          17970    0    0      0       0    431
    10.45.64.3    AK    2909975  36   13348  245614  0    45
    10.45.64.4    AK    228451   0    2192   60531   0    0
    10.45.64.7    AK    1119810  28   4610   15970   0    35
    10.45.64.11   AK    158206   0    335    52721   0    0
    10.45.64.12   AK    466173   0    14955  28006   3    0
    10.45.64.13   AK    287196   0    223    17277   0    0
    10.45.64.14   AK    2264292  13   20863  362708  0    23
    10.45.64.15   AK    46467    0    9      33894   0    0
    10.45.64.16   AK    12238    0    1396   3339    0    0
    10.45.64.17   AK    1058702  0    464    10578   0    0
    10.45.64.31   AK    1137817  0    46740  114016  12   0
    10.45.64.39   AK    519279   0    11019  44094   0    0
    10.45.64.45   AK    229911   0    363    297811  0    0
    10.45.64.47   AK    1257622  0    15731  410234  321  6041
    10.45.64.59   AK    186455   0    44     71189   0    0
    10.45.64.85   AK    4958     0    57     1069    0    0
    10.45.64.93   AK    604641   0    16530  89847   0    0
    10.45.64.103  AK    660575   0    10649  129808  0    0
    10.45.64.105  AK    407561   0    2046   28347   0    0
    10.45.64.112  AK    2007755  0    1135   58168   1    1
    10.45.64.116  AK    427696   0    1042   30081   0    0
    10.45.64.118  AK    275449   163  710    94141   0    170
    10.45.64.123  AK    461008   0    3118   29852   0    0
    10.45.64.125  AK    540511   0    52960  57523   49   0
    10.45.64.126  AK    686366   0    12579  57480   122  0
    10.45.65.7    A     959608   0    176    94920   0    0
    10.45.65.16   A     1392760  0    4483   157057  0    0
    10.45.65.110  A     694373   0    1530   44914   0    0
    10.45.65.112  A     1140897  0    2313   171205  0    0
    10.45.65.124  A     73846    5    6      24228   0    9
    10.45.86.39   A     823913   1    13     135427  0    0
    10.45.86.41   A     6872888  72   3228   1116750 0    0
    10.45.87.32   A     62967    0    64     33255   0    0
    10.45.87.45         2052604  1    0      225774  0    0
    10.45.87.50         2500669  3    3      134178  0    0
    10.45.87.127  A     527732   17   22     58020   0    39
    10.45.101.96        1883157  155  156    270789  0    151
    10.45.104.24  A     1837837  55   63     416284  0    77
    10.45.107.73  A     1851297  400  416    463753  0    430
    10.45.108.24  A     230952   0    0      33749   0    36
    10.45.113.6         10491356 168  191    253856  0    174
    10.45.113.106 A     638565   34   37     173265  0    40
    10.45.114.98        1370882  1619 1792   309940  0    1739
    10.45.138.66        479880   43   56     95577   0    47
    10.45.140.73        6036     510  551    1918    0    520
    10.45.140.74        5396418  39   46     336854  0    38
    10.45.141.85  A     450018   39   50     157585  0    85
    10.45.145.2   A     118843   10   15     19309   0    25
    10.45.145.73        3484464  1    0      153147  0    0
    10.45.153.10        714256   6    11     80712   0    0
    10.45.154.82  A     2355299  22   25     263636  0    44
    10.45.155.68        1849086  1    1      349722  0    0
    10.45.155.71  A     9089268  144  143    660457  1    152
    10.45.158.197       17618743 493  53     369921  0    0
    10.45.158.198 A     1019414  13   13     222440  0    25
    10.45.176.114 A     736022   32   46     88713   0    62
    10.45.176.119 A     2393601  38   47     167329  0    68
    10.45.177.68  A     17191899 22   24     201187  0    27
    10.45.182.75  A     615134   6    7      100995  0    13
    10.45.183.117       199726   131  145    52876   6    152
    10.45.183.204       2729641  8    10     851838  0    0
    10.45.212.82  A     2497321  18   23     213484  0    48
    10.45.229.81  A     1043941  268  2104   36517   1    0
    10.45.230.25  A     4560825  3132 18481  290819  0    0
    10.45.230.204       28514121 1    1      321299  0    0
    10.45.231.31  A     62246    16   9      12782   0    30
    10.45.234.197       265034   188  225    98664   0    153
    10.45.235.6   AK    140242   0    217    42778   0    0
    10.45.235.24  AK    213822   0    388    50485   0    0
    10.45.235.59  AK    1017759  16   897    121453  0    30
    10.45.235.89  AK    8066090  56   31899  637830  176  5630
    10.45.235.90  AK    3878916  727  4278   460048  0    709
    10.45.235.92  AK    15410232 151  169965 1030037 41   1784
    10.45.235.94  AK    344002   0    1394   3913608 0    0
    10.45.235.196 AK    2234     0    157    1323    0    0
    10.45.235.199 AK    2166     0    56     676     0    0
    10.45.235.200 AK    5279     0    220    2311    0    0
    10.45.235.203 AK    2966     0    234    1765    0    0
    10.45.235.206 AK    154701   0    3484   333     0    0
    10.45.235.208 AK    5076     0    378    3240    0    0
    10.45.238.75        4916805  196  262    942367  0    229
    10.45.241.101 A     2838055  0    740    303980  0    0
    10.45.242.72  A     81526    5    5      36884   0    11
    10.45.242.146 A     894737   21   25     85268   0    44
    10.45.243.69        1946622  2    3      64838   0    0
    10.45.249.6   A     621444   0    2461   75061   0    0
    10.45.249.34  A     180747   0    2260   128409  0    0
    10.45.251.37  A     455964   39   171    18996   0    0
    10.45.251.114 A     14208298 134  13794  946610  0    0
</artwork>
</section>

<section anchor="non-tcp-conntrack-supported-protocols-initiated-from-lan-to-wan"><name>Non-TCP conntrack-supported protocols initiated from LAN to WAN</name>

<artwork>Protocols included:

    UDP, ICMP, DCCP, SCTP, GRE

Active IPs:

    Active IPs with ECN signals:              420
    Active IP/dstport pairs with ECN signals: 24972

ECN flow packet count totals for active IPs:

    Direction CE      ECT(0)  ECT(1)
    --------- --      ------  ------
    From LAN  59      26692   28
    From WAN  2838929 9562002 6632561

ECN codepoint packet counts by client IP, with selected ports:
    (ports with '*' had &gt;100 ECT(0) marks)

                           ECT(0)  CE      ECT(1)  ECT(0) CE   ECT(1)
                           from    from    from    from   from from
    IP/Port                WAN     WAN     WAN     LAN    LAN  LAN
    -------                ---     ---     ---     ---    ---  ---
    10.45.10.0             0       0       0       201    0    0
    10.45.10.4             0       0       0       14     0    0
    10.45.10.5             0       0       0       20     0    0
    10.45.10.6             0       0       0       9      0    0
    10.45.10.7             0       0       0       8      0    0
    10.45.10.8             0       0       0       39     0    0
    10.45.10.11            0       0       0       8      0    0
    10.45.10.12            0       0       0       2      0    0
    10.45.10.42            0       0       0       6      0    0
    10.45.10.61            0       0       0       2      0    0
    10.45.10.70            0       0       0       44     0    0
    10.45.10.71            0       0       0       5      0    0
    10.45.10.73            0       0       0       7      0    0
    10.45.10.77            0       0       0       13     0    0
    10.45.10.81            0       0       0       10     0    0
    10.45.10.82            0       0       0       8      0    0
    10.45.10.83            0       0       0       3      0    0
    10.45.10.95            0       0       0       59     0    0
    10.45.10.96            0       0       0       39     0    0
    10.45.10.129           0       403     1       0      0    0
    10.45.10.196           0       0       0       80     0    0
    10.45.10.197           0       0       0       63     0    0
    10.45.10.201           0       0       0       3      0    0
    10.45.10.204           0       0       0       25     0    0
    10.45.10.227           0       0       0       40     0    0
    10.45.10.228           0       0       0       7      0    0
    10.45.10.244           0       0       0       14     0    0
    10.45.10.245           0       0       0       7      0    0
    10.45.64.3             0       0       0       100    0    0
    10.45.64.4             0       0       0       31     0    0
    10.45.64.6             0       0       0       2      0    0
    10.45.64.7             12      126     20      8      0    0
    10.45.64.10            0       0       0       29     0    0
    10.45.64.11            0       0       0       67     0    0
    10.45.64.12            0       0       0       6      0    0
    10.45.64.13            0       0       0       35     0    0
    10.45.64.14            0       0       0       121    0    0
    10.45.64.15            0       0       0       52     0    0
    10.45.64.16            0       0       0       18     0    0
    10.45.64.19            16      0       0       0      0    0
      udp:4500 (ipsec-na.. 11      0       0       0      0    0
    10.45.64.31            34129   2468    58304   27     0    0
      udp:37658            0       0       4346    0      0    0
    * udp:38129            24957   2468    15281   0      0    0
      udp:38884            0       0       10409   0      0    0
    * udp:40871            288     0       2269    0      0    0
    * udp:41621            3057    0       14609   0      0    0
    * udp:41744            171     0       61      0      0    0
      udp:43588            0       0       6746    0      0    0
      udp:45444            0       0       1292    0      0    0
    * udp:45465            866     0       0       0      0    0
      udp:45483            0       0       1838    0      0    0
    * udp:45522            4764    0       708     0      0    0
    10.45.64.39            0       0       0       75     0    0
    10.45.64.45            0       0       0       50     0    0
    10.45.64.47            0       0       0       11     0    0
    10.45.64.51            0       0       0       2      0    0
    10.45.64.59            56      1624    10      593    0    0
      udp:3478 (stun)      56      1624    10      0      0    0
    10.45.64.85            0       0       0       4      0    0
    10.45.64.86            7       434404  3       9      0    0
      udp:4400 (ds-srv)    0       29065   0       0      0    0
      udp:14757            0       97175   0       0      0    0
      udp:24173            0       35437   0       0      0    0
      udp:29493            0       120959  0       0      0    0
      udp:44495            0       41547   0       0      0    0
      udp:53678            0       109978  0       0      0    0
    10.45.64.89            7       50      0       4      0    0
    10.45.64.93            598     2971    341     75     0    0
    * udp:3478 (stun)      598     2971    341     0      0    0
    10.45.64.98            0       0       32780   0      0    0
      udp:6008             0       0       9234    0      0    0
      udp:7008 (afs3-upd.. 0       0       23546   0      0    0
    10.45.64.99            132     2094    73      0      0    0
      udp:3478 (stun)      0       3       0       0      0    0
    10.45.64.103           0       0       0       47     0    0
    10.45.64.104           70      293     31      1      0    0
    10.45.64.105           213     33440   0       7      0    0
    * udp:443 (https)      213     33440   0       0      0    0
    10.45.64.107           0       0       0       2      0    0
    10.45.64.108           0       0       0       1      0    0
    10.45.64.111           1       1       0       0      0    0
    10.45.64.112           0       421     0       48     0    0
    10.45.64.116           4       143     8       64     0    8
    10.45.64.118           0       0       0       77     0    0
    10.45.64.121           0       2107    0       0      0    0
      udp:38603            0       2100    0       0      0    0
    10.45.64.123           0       0       0       13     0    0
    10.45.64.124           6       0       0       0      0    0
      udp:443 (https)      6       0       0       0      0    0
    10.45.64.125           0       0       0       22     0    0
    10.45.64.126           1       10      0       37     0    0
    10.45.65.0             0       0       0       42     0    0
    10.45.65.1             0       0       0       45     0    0
    10.45.65.5             0       0       0       17     0    0
    10.45.65.7             0       0       0       30     0    0
    10.45.65.11            0       0       0       6      0    0
    10.45.65.16            1686    40141   36888   505    0    0
    * udp:3478 (stun)      1595    22049   4       0      0    0
      udp:26808            0       0       36805   0      0    0
      udp:62348            0       15738   0       0      0    0
    10.45.65.17            0       4       0       0      0    0
    10.45.65.66            0       17      0       94     0    0
      udp:3478 (stun)      0       17      0       0      0    0
    10.45.65.94            319     0       1       25     0    0
      udp:3478 (stun)      0       0       1       0      0    0
    10.45.65.95            0       0       0       8      0    0
    10.45.65.104           0       0       0       41     0    0
    10.45.65.107           12      77      2       5      0    0
    10.45.65.110           0       0       0       38     0    0
    10.45.65.112           39      1168    18      75     0    0
    10.45.65.122           2       5       0       0      0    0
    10.45.65.123           0       0       0       1      0    0
    10.45.65.124           0       0       0       11     0    0
    10.45.65.127           0       0       0       5      0    0
    10.45.75.90            0       0       0       1      0    0
    10.45.80.28            2       8       1       0      0    0
    10.45.80.79            4       7       0       2      0    0
    10.45.80.85            0       0       0       10     0    0
    10.45.80.99            0       0       0       11     0    0
    10.45.83.76            0       0       0       3      0    0
    10.45.83.80            28      51      11      0      0    0
    10.45.85.127           301     174     30747   68     0    0
    * udp:599 (acp)        222     174     45      0      0    0
      udp:6008             0       0       30702   0      0    0
    * udp:60001            49      0       0       65     0    0
    10.45.86.16            13      0       0       2      0    0
      udp:4500 (ipsec-na.. 8       0       0       0      0    0
      udp:51820 (wiregua.. 5       0       0       0      0    0
    10.45.86.36            0       0       0       4      0    0
    10.45.86.39            205     37619   107     50     0    0
      udp:29492            0       2512    0       0      0    0
      udp:64733            0       30711   0       0      0    0
    10.45.86.40            2       0       0       0      0    0
      udp:443 (https)      2       0       0       0      0    0
    10.45.86.43            0       11      0       532    0    0
    10.45.86.68            760     3528    614     325    0    0
      udp:80 (http)        0       2       0       0      0    0
    10.45.87.32            12      0       0       14     0    0
    10.45.87.44            709     4963    623     0      0    0
      udp:80 (http)        0       1       0       0      0    0
      udp:6881             3       1313    43      0      0    0
    10.45.87.45            0       0       0       185    0    0
    10.45.87.48            0       0       0       82     0    0
    10.45.87.50            3       0       9       68     0    0
      udp:4500 (ipsec-na.. 3       0       9       0      0    0
    10.45.87.103           0       0       0       2      0    0
    10.45.87.112           0       1       0       0      0    0
    10.45.87.113           0       0       0       33     0    0
    10.45.87.127           0       0       0       44     0    0
    10.45.92.74            31      0       1       2      0    0
    10.45.93.69            15      122     6       0      0    0
    10.45.93.75            361     2945    278     4      0    0
    10.45.93.79            0       0       0       8      0    0
    10.45.98.71            2       8       0       0      0    0
    10.45.98.72            0       1       0       40     0    0
      udp:3478 (stun)      0       1       0       0      0    0
    10.45.101.96           0       0       0       140    0    0
    10.45.101.100          0       0       0       12     0    0
    10.45.101.101          2       10      7       0      0    0
    10.45.101.103          21      21899   15      0      0    0
      udp:58479            0       21372   0       0      0    0
    10.45.101.104          0       0       10      33     0    0
    10.45.104.24           0       0       0       324    0    0
    10.45.104.104          16      72      2       60     0    0
    10.45.107.73           32      0       1       58     0    0
      udp:4500 (ipsec-na.. 32      0       1       0      0    0
    10.45.107.79           34      0       0       70     0    0
      udp:443 (https)      34      0       0       0      0    0
    10.45.107.81           0       4421    0       3      0    0
      udp:61094            0       4421    0       0      0    0
    10.45.108.3            0       0       0       1      0    0
    10.45.108.4            33      5079    90      1      0    0
      udp:33027            0       2978    0       0      0    0
    10.45.108.13           0       0       0       14     0    0
    10.45.108.24           799     5543    1059    117    0    0
    * udp:40211            107     0       0       0      0    0
    10.45.108.25           1       2       1       799    0    0
    10.45.108.66           0       0       0       0      0    1
    10.45.108.69           0       0       0       2      0    0
    10.45.108.71           28      12830   0       0      0    0
      udp:34665            0       12462   0       0      0    0
    10.45.108.75           0       0       6395176 38     0    0
      udp:6008             0       0       1755476 0      0    0
      udp:7008 (afs3-upd.. 0       0       1827173 0      0    0
      udp:8008 (http-alt)  0       0       740987  0      0    0
      udp:9008             0       0       809024  0      0    0
      udp:10008 (octopus)  0       0       380001  0      0    0
      udp:11008            0       0       578400  0      0    0
      udp:12008 (accurac.. 0       0       231619  0      0    0
      udp:13008            0       0       72496   0      0    0
    10.45.108.76           0       0       0       2      0    0
    10.45.108.77           0       0       0       31     0    0
    10.45.108.80           337     1566    173     10     0    0
    10.45.108.95           1       5       0       283    0    0
    10.45.108.126          0       0       0       12     0    0
    10.45.112.74           9       95      4       371    0    0
    10.45.112.102          0       0       0       29     0    0
    10.45.112.139          0       0       0       5      0    0
    10.45.112.154          0       0       0       4      0    0
    10.45.112.165          0       0       0       24     0    0
    10.45.112.172          6333    0       0       0      0    0
    * udp:443 (https)      6333    0       0       0      0    0
    10.45.112.216          0       0       0       2      0    0
    10.45.113.6            147184  0       0       136    0    0
    * udp:4500 (ipsec-na.. 147184  0       0       0      0    0
    10.45.113.7            453     0       10      52     0    0
    * udp:443 (https)      309     0       0       0      0    0
    * udp:4500 (ipsec-na.. 144     0       10      0      0    0
    10.45.113.9            0       0       0       60     0    0
    10.45.113.11           0       0       0       187    0    0
    10.45.113.27           0       0       0       1      0    0
    10.45.113.30           0       0       0       4      0    0
    10.45.113.33           0       0       0       2      0    0
    10.45.113.34           0       0       0       58     0    0
    10.45.113.35           0       0       0       6      0    0
    10.45.113.36           0       0       0       2      0    0
    10.45.113.66           1       11      0       0      0    0
    10.45.113.90           0       0       0       163    0    0
    10.45.113.94           17      62      2       0      0    0
    10.45.113.97           0       0       0       19     0    0
    10.45.113.99           11      76      12      15     0    0
    10.45.113.104          818     0       0       0      0    0
    * udp:4500 (ipsec-na.. 818     0       0       0      0    0
    10.45.113.106          0       0       0       10     0    0
    10.45.113.119          0       178     0       313    0    0
      udp:3478 (stun)      0       178     0       0      0    0
    10.45.113.122          36      0       0       0      0    0
      udp:4500 (ipsec-na.. 36      0       0       0      0    0
    10.45.113.124          0       0       0       201    0    0
    10.45.114.8            0       3       0       0      0    0
    10.45.114.10           0       0       0       3      0    0
    10.45.114.42           286     12      67      3      0    0
    * udp:51820 (wiregua.. 286     0       66      0      0    0
    10.45.114.98           0       0       0       10     0    0
    10.45.120.25           0       0       0       53     0    0
    10.45.120.34           0       0       0       12     0    0
    10.45.120.78           0       0       0       715    0    0
    10.45.122.51           686     28190   122     66     0    0
      udp:45622            0       5782    0       0      0    0
      udp:59437            0       17791   0       0      0    0
    10.45.124.31           1720    5946    16897   105    0    0
      udp:3478 (stun)      0       6       0       0      0    0
    * udp:50451            1720    0       15875   0      0    0
      udp:50919            0       2428    0       0      0    0
      udp:50996            0       0       1016    0      0    0
      udp:57403            0       1944    0       0      0    0
    10.45.124.43           0       0       0       12     0    0
    10.45.124.73           37      0       0       0      0    0
      udp:4500 (ipsec-na.. 37      0       0       0      0    0
    10.45.124.74           0       0       0       1      0    0
    10.45.124.89           0       0       0       2      0    0
    10.45.124.107          142     626895  83      0      0    0
      udp:24616            0       501142  0       0      0    0
      udp:51123            0       124060  0       0      0    0
    10.45.124.111          0       1538    166     0      0    0
      udp:4748             0       1491    166     0      0    0
    10.45.124.117          0       0       0       248    0    0
    10.45.125.97           0       0       0       2      0    0
    10.45.125.99           130     6235    29      1      0    0
      udp:8609 (canon-cp.. 0       3002    0       0      0    0
    10.45.125.104          0       0       0       3      0    0
    10.45.125.105          0       0       0       7      0    0
    10.45.136.82           0       0       0       1      0    0
    10.45.136.198          0       0       0       8      0    0
    10.45.136.199          68      3210    7       0      0    0
      udp:22312            0       2452    0       0      0    0
    10.45.136.200          0       44      1       0      0    0
    10.45.137.4            4603    0       0       1882   0    0
    * udp:443 (https)      4603    0       0       1882   0    0
    10.45.137.21           0       0       0       118    0    0
    10.45.137.27           4       0       0       63     0    0
    10.45.137.29           0       0       0       0      0    1
    10.45.137.46           9       154     0       6      0    0
      udp:443 (https)      9       0       0       0      0    0
    10.45.137.53           0       0       0       7      0    0
    10.45.137.55           0       0       1       37     0    0
    10.45.137.62           5       29      1       14     0    0
      udp:443 (https)      2       0       0       0      0    0
    10.45.137.119          16      203825  12      4      0    0
      udp:16772            0       55846   0       0      0    0
      udp:25135            0       24694   0       0      0    0
      udp:25476            0       66965   0       0      0    0
      udp:51123            0       54265   0       0      0    0
      udp:55430            0       1138    0       0      0    0
    10.45.137.123          2       4190    1       1      0    0
      udp:29363            0       3283    0       0      0    0
    10.45.138.52           3093    18938   0       0      0    0
    * udp:42420            3087    18871   0       0      0    0
    10.45.138.66           0       0       0       249    0    0
    10.45.138.88           43      107     10      0      0    0
    10.45.138.95           0       0       0       20     0    0
    10.45.140.0            0       0       0       84     0    0
    10.45.140.5            0       0       0       2      0    0
    10.45.140.28           0       0       0       1      0    0
    10.45.140.74           0       0       0       12     0    0
    10.45.140.81           0       0       0       26     0    0
    10.45.140.100          143     465     37      0      0    0
    10.45.140.103          0       0       0       16     0    0
    10.45.140.104          0       0       0       4      0    0
    10.45.140.109          0       0       0       2      0    0
    10.45.140.118          0       0       0       27     0    0
    10.45.140.121          0       7032    0       17     0    0
      udp:49710            0       1160    0       0      0    0
      udp:53984            0       2694    0       0      0    0
      udp:58704            0       1597    0       0      0    0
    10.45.140.122          0       3       0       0      0    0
    10.45.140.123          0       0       4       0      0    0
    10.45.140.127          0       0       0       15     0    0
    10.45.140.133          0       0       0       0      1    0
    10.45.140.169          0       0       0       59     0    0
    10.45.140.171          0       0       0       14     0    0
    10.45.141.2            91      0       0       12     0    0
      udp:443 (https)      91      0       0       0      0    0
    10.45.141.6            0       0       0       24     0    0
    10.45.141.14           0       0       0       2      0    0
    10.45.141.17           2       37      1       17     0    0
    10.45.141.19           0       0       0       2      0    0
    10.45.141.82           579     0       0       21     0    0
    * udp:443 (https)      579     0       0       0      0    0
    10.45.141.83           0       0       0       14     0    0
    10.45.141.84           0       0       0       90     0    0
    10.45.141.85           0       0       0       518    0    0
    10.45.141.86           0       0       0       6      0    0
    10.45.141.87           0       0       0       2      0    0
    10.45.141.103          0       0       0       57     0    0
    10.45.141.106          7       190     3947    1079   0    0
      udp:3478 (stun)      0       24      12      0      0    0
    * udp:5001 (commplex.. 0       0       0       1072   0    0
      udp:40208            0       0       3932    0      0    0
    10.45.141.125          0       0       0       2      0    0
    10.45.144.20           2       6       2       1      0    0
    10.45.144.43           0       0       0       3      0    0
    10.45.144.55           0       0       0       2      0    0
    10.45.144.68           0       0       0       363    0    0
    10.45.144.73           0       0       0       14     0    0
    10.45.144.75           0       0       3       51     0    0
    10.45.144.77           51      289     35      24     0    0
    10.45.144.105          413     0       11      1      0    0
    * udp:4500 (ipsec-na.. 413     0       11      0      0    0
    10.45.144.139          1496    0       0       0      0    0
    * udp:443 (https)      1496    0       0       0      0    0
    10.45.144.197          0       0       0       102    0    0
    10.45.145.2            0       0       0       15     0    0
    10.45.145.26           0       0       0       44     0    0
    10.45.145.39           2503039 0       0       11     0    0
      udp:443 (https)      4       0       0       0      0    0
    * udp:4500 (ipsec-na.. 2503035 0       0       0      0    0
    10.45.145.56           0       0       0       3      0    0
    10.45.145.72           0       0       0       32     0    0
    10.45.145.75           3024    0       0       0      0    0
    * udp:443 (https)      3024    0       0       0      0    0
    10.45.145.81           8691    107114  8245    292    0    0
      udp:80 (http)        0       2       0       0      0    0
    * udp:6881             355     8092    672     0      0    0
      udp:19517            0       1097    0       0      0    0
      udp:22784            0       3441    0       0      0    0
    * udp:25223            110     0       0       0      0    0
    * udp:37526            139     0       0       0      0    0
    * udp:40631            191     0       0       0      0    0
      udp:40990            0       33415   0       0      0    0
      udp:51820 (wiregua.. 0       3       0       0      0    0
    10.45.145.96           0       0       0       7      0    0
    10.45.145.98           0       0       0       3      0    0
    10.45.145.107          0       9       0       0      0    0
    10.45.145.109          0       0       0       9      35   0
    10.45.145.115          0       0       0       11     0    0
    10.45.146.66           52      88      7       26     0    0
    10.45.146.195          0       0       0       2      0    0
    10.45.146.200          1471    0       0       49     0    0
    * udp:4500 (ipsec-na.. 1471    0       0       0      0    0
    10.45.146.201          0       0       0       9      0    0
    10.45.153.10           0       0       0       33     0    0
    10.45.153.194          2       86      2       0      0    0
    10.45.154.6            0       0       0       9      0    0
    10.45.154.81           0       0       0       4      0    0
    10.45.154.82           0       0       0       140    0    0
    10.45.154.100          0       0       0       14     0    0
    10.45.154.105          0       0       0       17     0    0
    10.45.154.112          0       0       0       5      0    0
    10.45.154.113          1       88      2       3      0    0
    10.45.154.115          0       0       0       224    0    0
    10.45.155.12           0       0       0       11     0    0
    10.45.155.67           0       0       0       1      0    0
    10.45.155.68           0       0       0       237    0    0
    10.45.155.69           0       0       0       1      0    0
    10.45.155.71           0       0       0       246    0    0
    10.45.155.73           0       0       0       72     0    0
    10.45.155.74           0       1       0       0      0    0
      udp:3478 (stun)      0       1       0       0      0    0
    10.45.155.75           0       4       0       0      0    0
    10.45.155.76           0       1       0       0      0    0
    10.45.155.217          0       0       0       15     0    0
    10.45.155.229          4       42      6       48     0    0
    10.45.156.94           25      152     8       0      0    0
    10.45.156.105          0       5362    0       19     0    0
      udp:58796            0       5362    0       0      0    0
    10.45.156.127          0       0       0       22     0    0
    10.45.158.115          0       0       0       402    0    0
    10.45.158.124          0       0       0       4      0    0
    10.45.158.127          0       0       0       3      0    0
    10.45.158.195          0       1630    3       25     0    0
      udp:6881             0       1610    0       0      0    0
    10.45.158.197          0       0       0       82     0    0
    10.45.158.198          0       0       0       204    0    0
    10.45.158.204          0       0       0       118    0    0
    10.45.158.206          9       32      2       0      0    0
    10.45.176.114          0       0       0       68     0    0
    10.45.176.116          188     1702    191     1      0    0
    10.45.176.117          0       0       0       35     0    0
    10.45.176.119          9320    1028270 11302   218    0    0
      udp:6881             0       91498   83      0      0    0
    * udp:6900             322     0       0       0      0    0
      udp:8999 (bctp)      0       405853  3       0      0    0
    * udp:10556            741     0       0       0      0    0
      udp:11778            0       311705  0       0      0    0
    * udp:12111            274     0       0       0      0    0
      udp:21606            0       5678    0       0      0    0
      udp:23578            0       4281    0       0      0    0
      udp:24488            0       2140    0       0      0    0
      udp:35849            0       2632    0       0      0    0
    * udp:37758            212     721     0       0      0    0
      udp:40954            0       27113   0       0      0    0
    * udp:42012            380     26      101     0      0    0
      udp:48235            0       3182    0       0      0    0
    * udp:50321            2066    14226   5982    0      0    0
    * udp:50838            389     0       0       0      0    0
      udp:50884            0       0       2743    0      0    0
      udp:51413            39      1712    0       0      0    0
      udp:54457            0       3504    0       0      0    0
      udp:56769            0       23761   0       0      0    0
      udp:59025            0       3034    0       0      0    0
    * udp:60050            3000    3961    1478    0      0    0
      udp:60062            0       13672   0       0      0    0
      udp:64329            0       75590   0       0      0    0
    10.45.176.120          0       0       0       73     21   18
    10.45.176.206          37      689     3       34     0    0
      udp:3478 (stun)      37      685     3       0      0    0
    10.45.176.207          8       143     0       5      0    0
    10.45.176.209          12      88      1       11     0    0
    10.45.176.210          10      32      4       1      0    0
    10.45.176.214          25      8900    0       18     0    0
      udp:6672 (vision-s.. 23      8900    0       0      0    0
    10.45.176.224          1       0       0       114    0    0
    10.45.176.225          120     786     137     1      0    0
    10.45.176.226          0       0       0       4      0    0
    10.45.176.237          4       0       0       0      0    0
      udp:443 (https)      4       0       0       0      0    0
    10.45.177.66           9       213     8349    0      0    0
      udp:6672 (vision-s.. 0       0       8334    0      0    0
    10.45.177.68           12      64      8       124    0    0
    10.45.177.75           0       2       0       66     0    0
    10.45.177.197          0       2       1       0      0    0
    10.45.182.75           44      71      17      25     0    0
    10.45.182.85           41      2612    5024    0      0    0
      udp:45864            0       0       4985    0      0    0
    10.45.182.136          0       0       0       8      0    0
    10.45.183.117          0       0       0       15     0    0
    10.45.183.199          45      1579    0       8      0    0
      udp:3478 (stun)      45      1578    0       0      0    0
    10.45.183.204          0       9478    0       731    0    0
    * udp:4500 (ipsec-na.. 0       0       0       237    0    0
      udp:22885            0       9404    0       0      0    0
    10.45.183.205          0       0       1       3      0    0
      udp:4500 (ipsec-na.. 0       0       1       0      0    0
    10.45.183.209          3       1       0       280    0    0
    10.45.183.219          0       0       0       61     0    0
    10.45.203.6            0       0       0       2      0    0
    10.45.212.17           10472   25127   16430   0      0    0
    * udp:62503            10452   23528   16423   0      0    0
    10.45.212.27           0       0       0       1      0    0
    10.45.212.29           0       0       0       30     0    0
    10.45.212.51           0       0       0       2      0    0
    10.45.212.82           0       1       1       28     0    0
    10.45.212.84           0       0       0       2      0    0
    10.45.212.199          0       0       0       1      0    0
    10.45.212.202          0       0       0       4      0    0
    10.45.212.205          0       0       0       299    0    0
    10.45.212.207          0       0       0       85     0    0
    10.45.229.75           3       0       0       0      0    0
      udp:443 (https)      3       0       0       0      0    0
    10.45.229.78           6694314 0       0       113    0    0
    * udp:4500 (ipsec-na.. 6694314 0       0       0      0    0
    10.45.229.79           0       0       0       27     0    0
    10.45.229.81           0       0       0       3      0    0
    10.45.229.101          0       0       0       69     0    0
    10.45.229.104          128     525     128     0      0    0
    10.45.229.119          0       0       0       20     0    0
    10.45.230.20           0       0       0       1      0    0
    10.45.230.25           10      0       72      32     0    0
      udp:4500 (ipsec-na.. 10      0       72      0      0    0
    10.45.230.89           495     3537    296     4      0    0
    10.45.230.99           7       0       5       2      0    0
      udp:4500 (ipsec-na.. 7       0       5       0      0    0
    10.45.230.204          9       57      18      110    0    0
    10.45.230.207          18      33      1       1      0    0
    10.45.230.212          0       0       0       2      0    0
    10.45.230.223          0       0       0       3      0    0
    10.45.230.224          27927   93      13      0      0    0
    * udp:50323            322     0       0       0      0    0
    * udp:50361            128     0       0       0      0    0
    * udp:52065            409     0       0       0      0    0
    * udp:55236            257     0       0       0      0    0
    * udp:57072            142     0       0       0      0    0
    * udp:58494            170     0       0       0      0    0
    * udp:59465            160     0       0       0      0    0
    * udp:59659            445     0       0       0      0    0
    * udp:60874            129     0       0       0      0    0
    * udp:60898            102     0       0       0      0    0
    * udp:61122            302     0       0       0      0    0
    * udp:61312            137     0       0       0      0    0
    * udp:61669            124     0       0       0      0    0
    * udp:62889            24738   0       0       0      0    0
    * udp:63354            122     0       0       0      0    0
    * udp:63474            107     0       0       0      0    0
    10.45.230.226          0       0       0       3      0    0
    10.45.230.228          0       45      0       0      1    0
    10.45.230.229          682     21      3       0      0    0
    * udp:4500 (ipsec-na.. 682     0       0       0      0    0
    10.45.231.16           433     0       0       24     0    0
    * udp:4500 (ipsec-na.. 433     0       0       0      0    0
    10.45.231.21           40      256     81      0      0    0
    10.45.231.31           0       0       0       32     0    0
    10.45.231.53           2       46      0       0      0    0
    10.45.231.61           4151    0       1       13     0    0
    * udp:4500 (ipsec-na.. 4151    0       1       0      0    0
    10.45.231.80           0       0       0       6      0    0
    10.45.231.99           0       0       0       40     0    0
    10.45.231.102          0       0       0       11     0    0
    10.45.231.114          0       0       0       47     0    0
    10.45.233.16           0       0       0       55     0    0
    10.45.233.39           1       13      2       1      0    0
    10.45.233.41           0       0       3       4      0    0
    10.45.233.42           0       0       0       115    0    0
    10.45.233.47           0       0       0       1      0    0
    10.45.233.55           0       0       0       3      0    0
    10.45.234.197          320     0       11      2      0    0
    * udp:4500 (ipsec-na.. 320     0       11      0      0    0
    10.45.235.6            107     454     62      6      0    0
    10.45.235.11           250     0       0       0      0    0
    * udp:443 (https)      249     0       0       0      0    0
    10.45.235.13           0       0       0       4      0    0
    10.45.235.16           24      56      3       0      0    0
    10.45.235.19           0       0       0       3      0    0
    10.45.235.24           0       2       0       33     0    0
    10.45.235.25           2310    28152   68      17     0    0
    * udp:443 (https)      2214    0       0       0      0    0
      udp:6881             0       13339   0       0      0    0
      udp:31708            0       4595    0       0      0    0
      udp:51413            0       5367    0       0      0    0
      udp:52372            0       3975    0       0      0    0
    10.45.235.49           672     3165    14      0      0    0
    * udp:443 (https)      672     79      0       0      0    0
      udp:59418            0       3078    0       0      0    0
    10.45.235.52           23      0       0       0      0    0
      udp:4500 (ipsec-na.. 23      0       0       0      0    0
    10.45.235.59           0       0       0       58     0    0
    10.45.235.66           0       0       0       4      0    0
    10.45.235.89           165     2580    23      582    0    0
    * udp:3478 (stun)      165     2580    23      0      0    0
    10.45.235.90           0       0       0       332    0    0
    10.45.235.92           0       0       0       1007   0    0
    10.45.235.93           229     3272    306     13     0    0
    10.45.235.94           0       0       0       10     0    0
    10.45.238.75           0       0       0       1744   0    0
    10.45.238.104          0       2576    0       7      0    0
      udp:443 (https)      0       2576    0       0      0    0
    10.45.239.66           40      0       5       0      0    0
      udp:4500 (ipsec-na.. 40      0       5       0      0    0
    10.45.239.219          1       0       0       18     0    0
      udp:443 (https)      1       0       0       0      0    0
    10.45.240.86           0       0       0       5      0    0
    10.45.241.57           216     66079   437     0      0    0
      udp:4500 (ipsec-na.. 0       0       21      0      0    0
      udp:33522            0       37844   0       0      0    0
      udp:37859            0       27536   0       0      0    0
    10.45.241.94           0       0       0       44     0    0
    10.45.241.98           0       0       0       4      0    0
    10.45.241.101          68946   10      2       120    0    0
    * udp:4500 (ipsec-na.. 68942   0       0       0      0    0
    10.45.241.121          0       0       0       2      0    0
    10.45.242.72           0       0       0       4      0    0
    10.45.242.81           0       0       0       14     1    0
    10.45.242.144          0       0       0       5      0    0
    10.45.242.146          0       0       0       30     0    0
    10.45.242.161          143     134     2297    139    0    0
    * udp:4500 (ipsec-na.. 115     0       4       0      0    0
      udp:27032            0       78      2293    0      0    0
    10.45.243.13           13877   63      1       0      0    0
    * udp:20911            13853   0       0       0      0    0
    10.45.243.41           12      0       0       14     0    0
      udp:443 (https)      12      0       0       0      0    0
    10.45.243.69           0       0       0       66     0    0
    10.45.243.71           0       28      0       2      0    0
      udp:80 (http)        0       28      0       0      0    0
    10.45.243.109          0       2008    0       7      0    0
      udp:41697            0       2002    0       0      0    0
    10.45.248.33           2       8       0       10     0    0
      udp:3478 (stun)      2       8       0       0      0    0
    10.45.248.94           0       0       0       11     0    0
    10.45.248.118          0       0       0       2      0    0
    10.45.249.6            0       0       0       1502   0    0
    10.45.249.34           25      0       0       154    0    0
      udp:443 (https)      25      0       0       0      0    0
    10.45.249.99           68      558     88      0      0    0
    10.45.249.104          0       0       0       7      0    0
      udp:4500 (ipsec-na.. 0       0       0       6      0    0
    10.45.250.89           0       0       0       5      0    0
    10.45.251.37           0       0       0       19     0    0
    10.45.251.110          9       72      1       0      0    0
    10.45.251.119          0       0       0       23     0    0
    10.45.253.59           0       0       0       1      0    0
      udp:4500 (ipsec-na.. 0       0       0       1      0    0
    10.45.253.61           0       0       0       53     0    0
    10.45.253.84           121     0       0       16     0    0
    * udp:443 (https)      121     0       0       0      0    0
    10.45.253.93           0       0       0       4      0    0
    10.45.253.100          0       0       0       142    0    0
    10.45.253.121          0       0       0       2      0    0
    10.45.254.94           0       0       0       12     0    0
    10.45.255.90           1       125     0       0      0    0
    10.45.255.97           0       0       0       36     0    0

ECN codepoint packet counts for selected ports:

                           ECT(0)  CE      ECT(1)  ECT(0) CE   ECT(1)
                           from    from    from    from   from from
    Port                   WAN     WAN     WAN     LAN    LAN  LAN
    ----                   ---     ---     ---     ---    ---  ---
    icmp:port-unreachable  6632    40795   3539    404    0    0
    icmp:host-unreachable  1171    2575    43      22990  0    0
    icmp:ttl-zero-during.. 65      2       66      0      0    0
    icmp:network-unreach.. 321     4       0       0      0    0
    ipencap:0              0       0       0       1      0    0
    udp:53 (domain)        0       403     1       0      0    0
    udp:80 (http)          0       33      0       0      0    0
    udp:443 (https)        20006   36095   0       1882   0    0
    udp:599 (acp)          238     261     59      0      0    0
    udp:1024-3457 [81]     100     618     9       34     59   28
    udp:3478 (stun)        2498    31725   394     0      0    0
    udp:3553-4492 [19]     1       29449   0       0      0    0
    udp:4500 (ipsec-nat-t) 9422229 0       151     244    0    0
    udp:4548-51819 [8177]  62692   2291117 6604184 1072   0    0
    udp:51820 (wireguard)  291     3       66      0      0    0
    udp:51821-65535 [9371] 45758   405849  24049   65     0    0
</artwork>
</section>
</section>

<section anchor="iana-considerations"><name>IANA Considerations</name>
<t>This document has no IANA actions.</t>
</section>

<section anchor="security-considerations"><name>Security Considerations</name>
<t>There are no known security considerations introduced by this note.</t>
</section>

<section anchor="affiliation"><name>Affiliation</name>
<t>The author wrote the script used to deploy fq_codel to the two backhaul subnets
mentioned in <xref target="rfc3168-aqm-activity"></xref>.</t>
</section>

<section anchor="acknowledgements"><name>Acknowledgements</name>
<t>Thanks go to:</t>

<ul>
<li>Adam Pribyl, for gathering data at the FreeNet gateway</li>
<li>Jonathan Morton and Rodney Grimes, for helping to analyze the results</li>
<li>FreeNet Liberec, for allowing access for data collection</li>
</ul>
</section>

</middle>

<back>
<references><name>Informative References</name>
<reference anchor="IPTABLES-ECN" target="https://github.com/heistp/iptables-ecn/">
  <front>
    <title>iptables-ecn GitHub Repository</title>
    <author initials="P.G." surname="Heist"></author>
    <date></date>
  </front>
</reference>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1349.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2481.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3168.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3540.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3948.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6040.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8290.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8311.xml"/>
<reference anchor="WIREGUARD" target="https://www.wireguard.com">
  <front>
    <title>WireGuard web site</title>
    <author></author>
    <date></date>
  </front>
</reference>
</references>

</back>

</rfc>
