<?xml version="1.0" encoding="us-ascii"?>
  <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
  <!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.2.14 -->

<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
<!ENTITY RFC2544 SYSTEM "https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2544.xml">
<!ENTITY RFC8174 SYSTEM "https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml">
]>

<?rfc toc="yes"?>
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes"?>

<rfc ipr="trust200902" docName="draft-vpolak-bmwg-plrsearch-03" category="info">

  <front>
    <title>Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)</title>

    <author initials="M." surname="Konstantynowicz" fullname="Maciek Konstantynowicz" role="editor">
      <organization>Cisco Systems</organization>
      <address>
        <email>mkonstan@cisco.com</email>
      </address>
    </author>
    <author initials="V." surname="Polak" fullname="Vratko Polak" role="editor">
      <organization>Cisco Systems</organization>
      <address>
        <email>vrpolak@cisco.com</email>
      </address>
    </author>

    <date year="2020" month="March" day="06"/>

    <area>ops</area>
    <workgroup>Benchmarking Working Group</workgroup>
    <keyword>Internet-Draft</keyword>

    <abstract>


<t>This document addresses challenges while applying methodologies
described in <xref target="RFC2544"></xref> to benchmarking software based NFV (Network
Function Virtualization) data planes over an extended period of time,
sometimes referred to as "soak testing". Packet throughput search
approach proposed by this document assumes that system under test is
probabilistic in nature, and not deterministic.</t>



    </abstract>


  </front>

  <middle>


<section anchor="motivation" title="Motivation">

<t>Network providers are interested in throughput a networking system can
sustain.</t>

<t><xref target="RFC2544"></xref> assumes loss ratio is given by a deterministic function of
offered load. But NFV software systems are not deterministic enough.
This makes deterministic algorithms (such as Binary Search per <xref target="RFC2544"></xref>
and <xref target="draft-vpolak-mkonstan-bmwg-mlrsearch"></xref> with single trial) to return
results, which when repeated show relatively high standard deviation,
thus making it harder to tell what "the throughput" actually is.</t>

<t>We need another algorithm, which takes this indeterminism into account.</t>

</section>
<section anchor="relation-to-rfc2544" title="Relation To RFC2544">

<t>The aim of this document is to become an extension of <xref target="RFC2544"></xref> suitable
for benchmarking networking setups such as software based NFV systems.</t>

</section>
<section anchor="terms-and-assumptions" title="Terms And Assumptions">

<t>Due to the indeterministic nature of certain NFV systems that are the
targetted by PLRsearch algorithm, existing network benchmarking terms
are explicated and a number of new terms and assumptions are introduced.</t>

<section anchor="device-under-test" title="Device Under Test">

<t>In software networking, "device" denotes a specific piece of software
tasked with packet processing. Such device is surrounded with other
software components (such as operating system kernel). It is not
possible to run devices without also running the other components, and
hardware resources are shared between both.</t>

<t>For purposes of testing, the whole set of hardware and software
components is called "system under test" (SUT). As SUT is the part of
the whole test setup performance of which can be measured by <xref target="RFC2544"></xref>
methods, this document uses SUT instead of <xref target="RFC2544"></xref> DUT.</t>

<t>Device under test (DUT) can be re-introduced when analysing test results
using whitebox techniques, but that is outside the scope of this document.</t>

</section>
<section anchor="system-under-test" title="System Under Test">

<t>System under test (SUT) is a part of the whole test setup whose
performance is to be benchmarked. The complete methodology contains
other parts, whose performance is either already established, or not
affecting the benchmarking result.</t>

</section>
<section anchor="sut-configuration" title="SUT Configuration">

<t>Usually, system under test allows different configurations, affecting
its performance. The rest of this document assumes a single
configuration has been chosen.</t>

</section>
<section anchor="sut-setup" title="SUT Setup">

<t>Similarly to <xref target="RFC2544"></xref>, it is assumed that the system under test has
been updated with all the packet forwarding information it needs, before
the trial measurements (see below) start.</t>

</section>
<section anchor="network-traffic" title="Network Traffic">

<t>Network traffic is a type of interaction between system under test and
the rest of the system (traffic generator), used to gather information
about the system under test performance. PLRsearch is applicable only to
areas where network traffic consists of packets.</t>

</section>
<section anchor="packet" title="Packet">

<t>Unit of interaction between traffic generator and the system under test.
Term "packet" is used also as an abstraction of Ethernet frames.</t>

<section anchor="packet-offered" title="Packet Offered">

<t>Packet can be offered, which means it is sent from traffic generator
to the system under test.</t>

<t>Each offered packet is assumed to become received or lost in a short
time.</t>

</section>
<section anchor="packet-received" title="Packet Received">

<t>Packet can be received, which means the traffic generator verifies it
has been processed. Typically, when it is succesfully sent from the
system under test to traffic generator.</t>

<t>It is assumed that each received packet has been caused by an offered
packet, so the number of packets received cannot be larger than the
number of packets offered.</t>

</section>
<section anchor="packet-lost" title="Packet Lost">

<t>Packet can be lost, which means sent but not received in a timely
manner.</t>

<t>It is assumed that each lost packet has been caused by an offered
packet, so the number of packets lost cannot be larger than the number
of packets offered.</t>

<t>Usually, the number of packets lost is computed as the number of packets
offered, minus the number of packets received.</t>

</section>
<section anchor="other-packets" title="Other Packets">

<t>PLRsearch is not considering other packet behaviors known from
networking (duplicated, reordered, greatly delayed), assuming the test
specification reclassifies those behaviors to fit into the first three
categories.</t>

</section>
</section>
<section anchor="traffic-profile" title="Traffic Profile">

<t>Usually, the performance of the system under test depends on a "type" of
a particular packet (for example size), and "composition" if the network
traffic consists of a mixture of different packet types.</t>

<t>Also, some systems under test contain multiple "ports" packets can be
offered to and received from.</t>

<t>All such qualities together (but not including properties of trial
measurements) are called traffic profile.</t>

<t>Similarly to system under test configuration, this document assumes only
one traffic profile has been chosen for a particular test.</t>

</section>
<section anchor="traffic-generator" title="Traffic Generator">

<t>Traffic generator is the part of the whole test setup, distinct from the
system under test, responsible both for offering packets in a highly
predictable manner (so the number of packets offered is known), and for
counting received packets in a precise enough way (to distinguish lost
packets from tolerably delayed packets).</t>

<t>Traffic generator must offer only packets compatible with the traffic
profile, and only count similarly compatible packets as received.</t>

<t>Criteria defining which received packets are compatible are left
for test specification to decide.</t>

</section>
<section anchor="offered-load" title="Offered Load">

<t>Offered load is an aggregate rate (measured in packets per second) of
network traffic offered to the system under test, the rate is kept
constant for the duration of trial measurement.</t>

</section>
<section anchor="trial-measurement" title="Trial Measurement">

<t>Trial measurement is a process of stressing (previously setup) system
under test by offering traffic of a particular offered load, for a
particular duration.</t>

<t>After that, the system has a short time to become idle, while the
traffic generator decides how many packets were lost.</t>

<t>After that, another trial measurement (possibly with different offered
load and duration) can be immediately performed. Traffic generator
should ignore received packets caused by packets offered in previous
trial measurements.</t>

</section>
<section anchor="trial-duration" title="Trial Duration">

<t>Duration for which the traffic generator was offering packets at
constant offered load.</t>

<t>In theory, care has to be taken to ensure the offered load and trial
duration predict integer number of packets to offer, and that the
traffic generator really sends appropriate number of packets within
precisely enough timed duration. In practice, such consideration do not
change PLRsearch result in any significant way.</t>

</section>
<section anchor="packet-loss" title="Packet Loss">

<t>Packet loss is any quantity describing a result of trial measurement.</t>

<t>It can be loss count, loss rate or loss ratio. Packet loss is zero (or
non-zero) if either of the three quantities are zero (or non-zero,
respecively).</t>

<section anchor="loss-count" title="Loss Count">

<t>Number of packets lost (or delayed too much) at a trial measurement by
the system under test as determined by packet generator. Measured in
packets.</t>

</section>
<section anchor="loss-rate" title="Loss Rate">

<t>Loss rate is computed as loss count divided by trial duration. Measured
in packets per second.</t>

</section>
<section anchor="loss-ratio" title="Loss Ratio">

<t>Loss ratio is computed as loss count divided by number of packets
offered. Measured as a real (in practice rational) number between zero
or one (including).</t>

</section>
</section>
<section anchor="trial-order-independent-system" title="Trial Order Independent System">

<t>Trial order independent system is a system under test, proven (or just
assumed) to produce trial measurement results that display trial order
independence.</t>

<t>That means when a pair of consequent trial measurements are performed,
the probability to observe a pair of specific results is the same, as
the probability to observe the reversed pair of results whe performing
the reversed pair of consequent measurements.</t>

<t>PLRsearch assumes the system under test is trial order independent.</t>

<t>In practice, most system under test are not entirely trial order
independent, but it is not easy to devise an algorithm taking that into
account.</t>

</section>
<section anchor="trial-measurement-result-distribution" title="Trial Measurement Result Distribution">

<t>When a trial order independent system is subjected to repeated trial
measurements of constant duration and offered load, Law of Large Numbers
implies the observed loss count frequencies will converge to a specific
probability distribution over possible loss counts.</t>

<t>This probability distribution is called trial measurement result
distribution, and it depends on all properties fixed when defining it.
That includes the system under test, its chosen configuration, the
chosen traffic profile, the offered load and the trial duration.</t>

<t>As the system is trial order independent, trial measurement result
distribution does not depend on results of few initial trial
measurements, of any offered load or (finite) duration.</t>

</section>
<section anchor="average-loss-ratio" title="Average Loss Ratio">

<t>Probability distribution over some (finite) set of states enables
computation of probability-weighted average of any quantity evaluated on
the states (called the expected value of the quantity).</t>

<t>Average loss ratio is simply the expected value of loss ratio for a
given trial measurement result distribution.</t>

</section>
<section anchor="duration-independent-system" title="Duration Independent System">

<t>Duration independent system is a trial order independent system, whose
trial measurement result distribution is proven (or just assumed) to
display practical independence from trial duration. See definition of
trial duration for discussion on practical versus theoretical.</t>

<t>The only requirement is for average loss ratio to be independent of
trial duration.</t>

<t>In theory, that would necessitate each trial measurement result
distribution to be a binomial distribution. In practice, more
distributions are allowed.</t>

<t>PLRsearch assumes the system under test is duration independent, at
least for trial durations typically chosen for trial measurements
initiated by PLRsearch.</t>

</section>
<section anchor="load-regions" title="Load Regions">

<t>For a duration independent system, trial measurement result distribution
depends only on offered load.</t>

<t>It is convenient to name some areas of offered load space by possible
trial results.</t>

<section anchor="zero-loss-region" title="Zero Loss Region">

<t>A particular offered load value is said to belong to zero loss region,
if the probability of seeing non-zero loss trial measurement result is
exactly zero, or at least practically indistinguishable from zero.</t>

</section>
<section anchor="guaranteed-loss-region" title="Guaranteed Loss Region">

<t>A particular offered load value is said to belong to guaranteed loss
region, if the probability of seeing zero loss trial measurement result
(for non-negligible count of packets offered) is exactly zero, or at
least practically indistinguishable from zero.</t>

</section>
<section anchor="non-deterministic-region" title="Non-Deterministic Region">

<t>A particular offered load value is said to belong to non-deterministic
region, if the probability of seeing zero loss trial measurement result
(for non-negligible count of packets offered) is practically
distinguishable from both zero and one.</t>

</section>
<section anchor="normal-region-ordering" title="Normal Region Ordering">

<t>Although theoretically the three regions can be arbitrary sets, this
document assumes they are intervals, where zero loss region contains
values smaller than non-deterministic region, which in turn contains
values smaller than guaranteed loss region.</t>

</section>
</section>
<section anchor="deterministic-system" title="Deterministic System">

<t>A hypothetical duration independent system with normal region ordering,
whose non-deterministic region is extremely narrow (only present due to
"practical distinguishibility" and cases when the expected number of
packets offered is not and integer).</t>

<t>A duration independent system which is not deterministic is called non-
deterministic system.</t>

</section>
<section anchor="througphput" title="Througphput">

<t>Throughput is the highest offered load provably causing zero packet loss
for trial measurements of duration at least 60 seconds.</t>

<t>For duration independent systems with normal region ordering, the
throughput is the highest value within the zero loss region.</t>

</section>
<section anchor="deterministic-search" title="Deterministic Search">

<t>Any algorithm that assumes each measurement is a proof of the offered
load belonging to zero loss region (or not) is called deterministic
search.</t>

<t>This definition includes algorithms based on "composite measurements"
which perform multiple trial measurements, somehow re-classifying
results pointing at non-deterministic region.</t>

<t>Binary Search is an example of deterministic search.</t>

<t>Single run of a deterministic search launched against a deterministic
system is guaranteed to find the throughput with any prescribed
precision (not better than non-deterministic region width).</t>

<t>Multiple runs of a deterministic search launched against a non-
deterministic system can return varied results within non-deterministic
region. The exact distribution of deterministic search results depends
on the algorithm used.</t>

</section>
<section anchor="probabilistic-search" title="Probabilistic Search">

<t>Any algorithm which performs probabilistic computations based on
observed results of trial measurements, and which does not assume that
non-deterministic region is practically absent, is called probabilistic
search.</t>

<t>A probabilistic search algorithm, which would assume that non-
deterministic region is practically absent, does not really need to
perform probabilistic computations, so it would become a deterministic
search.</t>

<t>While probabilistic search for estimating throughput is possible, it
would need a careful model for boundary between zero loss region and
non-deterministic region, and it would need a lot of measurements of
almost surely zero loss to reach good precision.</t>

</section>
<section anchor="loss-ratio-function" title="Loss Ratio Function">

<t>For any duration independent system, the average loss ratio depends only
on offered load (for a particular test setup).</t>

<t>Loss ratio function is the name used for the function mapping offered
load to average loss ratio.</t>

<t>This function is initially unknown.</t>

</section>
<section anchor="target-loss-ratio" title="Target Loss Ratio">

<t>Input parameter of PLRsearch. The average loss ratio the output of
PLRsearch aims to achieve.</t>

</section>
<section anchor="critical-load" title="Critical Load">

<t>Aggregate rate of network traffic, which would lead to average loss
ratio exactly matching target loss ratio, if used as the offered load
for infinite many trial measurement.</t>

</section>
<section anchor="critical-load-estimate" title="Critical Load Estimate">

<t>Any quantitative description of the possible critical load PLRsearch is
able to give after observing finite amount of trial measurements.</t>

</section>
<section anchor="fitting-function" title="Fitting Function">

<t>Any function PLRsearch uses internally instead of the unknown loss ratio
function. Typically chosen from small set of formulas (shapes) with few
parameters to tweak.</t>

</section>
<section anchor="shape-of-fitting-function" title="Shape of Fitting Function">

<t>Any formula with few undetermined parameters.</t>

</section>
<section anchor="parameter-space" title="Parameter Space">

<t>A subset of Real Coordinate Space. A point of parameter space is a
vector of real numbers. Fitting function is defined by shape (a formula
with parameters) and point of parameter space (specifying values for the
parameters).</t>

</section>
</section>
<section anchor="abstract-algorithm" title="Abstract Algorithm">

<section anchor="high-level-description" title="High level description">

<t>PLRsearch accepts some input arguments, then iteratively performs trial
measurements at varying offered loads (and durations), and returns some
estimates of critical load.</t>

<t>PLRsearch input arguments form three groups.</t>

<t>First group has a single argument: measurer. This is a callback
(function) accepting offered load and duration, and returning the
measured loss count.</t>

<t>Second group consists of load related arguments required for measurer to
work correctly, typically minimal and maximal load to offer. Also,
target loss ratio (if not hardcoded) is a required argument.</t>

<t>Third group consists of time related arguments. Typically the duration
for the first trial measurement, duration increment per subsequent trial
measurement, and total time for search. Some PLRsearch implementation may
use estimation accuracy parameters as an exit condition instead of total
search time.</t>

<t>The returned quantities should describe the final (or best) estimate of
critical load. Implementers can chose any description that suits their
users, typically it is average and standard deviation, or lower and
upper boundary.</t>

</section>
<section anchor="main-ideas" title="Main Ideas">

<t>The search tries to perform measurements at offered load close to the
critical load, because measurement results at offered loads far from the
critical load give less information on precise location of the critical
load. As virtually every trial measurement result alters the estimate of
the critical load, offered loads vary as they approach the critical
load.</t>

<t>The only quantity of trial measurement result affecting the computation
is loss count. No latency (or other information) is taken into account.</t>

<t>PLRsearch uses Bayesian Inference, computed using numerical integration,
which takes long time to get reliable enough results. Therefore it takes
some time before the most recent measurement result starts affecting
subsequent offered loads and critical rate estimates.</t>

<t>During the search, PLRsearch spawns few processes that perform numerical
computations, the main process is calling the measurer to perform trial
measurements, without any significant delays between them. The durations
of the trial measurements are increasing linearly, as higher number of
trial measurement results take longer to process.</t>

<section anchor="trial-durations" title="Trial Durations">

<t><xref target="RFC2544"></xref> motivates the usage of at least 60 second duration by the idea
of the system under test slowly running out of resources (such as memory
buffers).</t>

<t>Practical results when measuring NFV software systems show that relative
change of trial duration has negligible effects on average loss ratio,
compared to relative change in offered load.</t>

<t>While the standard deviation of loss ratio usually shows some effects of
trial duration, they are hard to model. So PLRsearch assumes SUT
is duration independent, and chooses trial durations only based on
numeric integration requirements.</t>

</section>
<section anchor="target-loss-ratio-1" title="Target Loss Ratio">

<t>(TODO: Link to why we think 1e-7 is acceptable loss ratio.)</t>

</section>
</section>
<section anchor="plrsearch-building-blocks" title="PLRsearch Building Blocks">

<t>Here we define notions used by PLRsearch which are not applicable to
other search methods, nor probabilistic systems under test in general.</t>

<section anchor="bayesian-inference" title="Bayesian Inference">

<t>PLRsearch uses a fixed set of fitting function shapes,
and uses Bayesian inference to track posterior distribution
on each fitting function parameter space.</t>

<t>Specifically, the few parameters describing a fitting function
become the model space. Given a prior over the model space, and trial
duration results, a posterior distribution is computed, together
with quantities describing the critical load estimate.</t>

<t>Likelihood of a particular loss count is computed using Poisson distribution
of average loss rate given by the fitting function (at specific point of
parameter space).</t>

<t>Side note: Binomial Distribution is a better fit compared to Poisson
distribution (acknowledging that the number of packets lost cannot be
higher than the number of packets offered), but the difference tends to
be relevant only in high loss region. Using Poisson distribution lowers
the impact of measurements in high loss region, thus helping the
algorithm to converge towards critical load faster.</t>

</section>
<section anchor="iterative-search" title="Iterative Search">

<t>The idea PLRsearch is to iterate trial measurements, using Bayesian
inference to compute both the current estimate of the critical load and
the next offered load to measure at.</t>

<t>The required numerical computations are done in parallel with the trial
measurements.</t>

<t>This means the result of measurement "n" comes as an (additional) input
to the computation running in parallel with measurement "n+1", and the
outputs of the computation are used for determining the offered load for
measurement "n+2".</t>

<t>Other schemes are possible, aimed to increase the number of measurements
(by decreasing their duration), which would have even higher number of
measurements run before a result of a measurement affects offered load.</t>

</section>
<section anchor="fitting-functions" title="Fitting Functions">

<t>To make the space of possible loss ratio functions more tractable
the algorithm uses only few fitting function shapes for its predicitons.
As the search algorithm needs to evaluate the function also far away
from the critical load, the fitting function have to be reasonably behaved
for every positive offered load, specifically cannot cannot predict
non-positive packet loss ratio.</t>

</section>
<section anchor="measurement-impact" title="Measurement Impact">

<t>Results from trials far from the critical load are likely to affect
the critical load estimate negatively, as the fitting functions do not
need to be good approximations there. This is true mainly for guaranteed loss
region, as in zero loss region even badly behaved fitting function
predicts loss count to be "almost zero", so seeing a measurement
confirming the loss has been zero indeed has small impact.</t>

<t>Discarding some results, or "suppressing" their impact with ad-hoc
methods (other than using Poisson distribution instead of binomial) is
not used, as such methods tend to make the overall search unstable. We
rely on most of measurements being done (eventually) near the critical load,
and overweighting far-off measurements (eventually) for well-behaved
fitting functions.</t>

</section>
<section anchor="fitting-function-coefficients-distribution" title="Fitting Function Coefficients Distribution">

<t>To accomodate systems with different behaviours, a fitting function is
expected to have few numeric parameters affecting its shape (mainly
affecting the linear approximation in the critical region).</t>

<t>The general search algorithm can use whatever increasing fitting
function, some specific functions are described later.</t>

<t>It is up to implementer to chose a fitting function and prior
distribution of its parameters. The rest of this document assumes each
parameter is independently and uniformly distributed over a common
interval. Implementers are to add non-linear transformations into their
fitting functions if their prior is different.</t>

</section>
<section anchor="exit-condition" title="Exit Condition">

<t>Exit condition for the search is either the standard deviation of the
critical load estimate becoming small enough (or similar), or overal
search time becoming long enough.</t>

<t>The algorithm should report both average and standard deviation for its
critical load posterior.</t>

</section>
<section anchor="integration" title="Integration">

<t>The posterior distributions for fitting function parameters are not be
integrable in general.</t>

<t>The search algorithm utilises the fact that trial measurement takes some
time, so this time can be used for numeric integration (using suitable
method, such as Monte Carlo) to achieve sufficient precision.</t>

</section>
<section anchor="optimizations" title="Optimizations">

<t>After enough trials, the posterior distribution will be concentrated in
a narrow area of the parameter space. The integration method should take
advantage of that.</t>

<t>Even in the concentrated area, the likelihood can be quite small, so the
integration algorithm should avoid underflow errors by some means,
for example by tracking the logarithm of the likelihood.</t>

</section>
<section anchor="offered-load-selection" title="Offered Load Selection">

<t>The simplest rule is to set offered load for next trial measurememnt
equal to the current average (both over posterio and over
fitting function shapes) of the critical load estimate.</t>

<t>Contrary to critical load estimate computation, heuristic algorithms
affecting offered load selection do not introduce instability,
and can help with convergence speed.</t>

</section>
<section anchor="trend-analysis" title="Trend Analysis">

<t>If the reported averages follow a trend (maybe without reaching equilibrium),
average and standard deviation COULD refer to the equilibrium estimates
based on the trend, not to immediate posterior values.</t>

<t>But such post-processing is discouraged, unless a clear reason
for the trend is known. Frequently, presence of such a trend is a sign
of some of PLRsearch assumption being violated (usually trial order
independence or duration independence).</t>

<t>It is RECOMMENDED to report any trend quantification together with
direct critical load estimate, so users can draw their own conclusion.
Alternatively, trend analysis may be a part of exit conditions,
requiring longer searches for systems displaying trends.</t>

</section>
</section>
</section>
<section anchor="known-implementations" title="Known Implementations">

<t>The only known working implementation of PLRsearch is in Linux
Foundation FD.io CSIT open-source project <xref target="FDio-CSIT-PLRsearch"></xref>.</t>

<section anchor="fdio-csit-implementation-specifics" title="FD.io CSIT Implementation Specifics">

<t>The search receives min_rate and max_rate values, to avoid measurements
at offered loads not supporeted by the traffic generator.</t>

<t>The implemented tests cases use bidirectional traffic. The algorithm
stores each rate as bidirectional rate (internally, the algorithm is
agnostic to flows and directions, it only cares about overall counts of
packets sent and packets lost), but debug output from traffic generator
lists unidirectional values.</t>

<section anchor="measurement-delay" title="Measurement Delay">

<t>In a sample implemenation in FD.io CSIT project, there is roughly 0.5
second delay between trials due to restrictons imposed by packet traffic
generator in use (T-Rex).</t>

<t>As measurements results come in, posterior distribution computation
takes more time (per sample), although there is a considerable constant
part (mostly for inverting the fitting functions).</t>

<t>Also, the integrator needs a fair amount of samples to reach the region
the posterior distribution is concentrated at.</t>

<t>And of course, speed of the integrator depends on computing power of the
CPUs the algorithm is able to use.</t>

<t>All those timing related effects are addressed by arithmetically
increasing trial durations with configurable coefficients (currently 5.1
seconds for the first trial, each subsequent trial being 0.1 second
longer).</t>

</section>
<section anchor="rounding-errors-and-underflows" title="Rounding Errors and Underflows">

<t>In order to avoid them, the current implementation tracks natural
logarithm (instead of the original quantity) for any quantity which is
never negative. Logarithm of zero is minus infinity (not supported by
Python), so special value "None" is used instead. Specific functions for
frequent operations (such as "logarithm of sum of exponentials") are
defined to handle None correctly.</t>

</section>
<section anchor="fitting-functions-1" title="Fitting Functions">

<t>Current implementation uses two fitting functions. In general, their
estimates for critical rate differ, which adds a simple source of
systematic error, on top of posterior dispersion reported by integrator.
Otherwise the reported stdev of critical rate estimate is
unrealistically low.</t>

<t>Both functions are not only increasing, but also convex (meaning the
rate of increase is also increasing).</t>

<t>As Primitive Function to any positive function is an increasing
function, and Primitive Function to any increasing function is convex
function; both fitting functions were constructed as double Primitive
Function to a positive function (even though the intermediate increasing
function is easier to describe).</t>

<t>As not any function is integrable, some more realistic functions
(especially with respect to behavior at very small offered loads) are
not easily available.</t>

<t>Both fitting functions have a "central point" and a "spread", varied by
simply shifting and scaling (in x-axis, the offered load direction) the
function to be doubly integrated. Scaling in y-axis (the loss rate
direction) is fixed by the requirement of transfer rate staying nearly
constant in very high offered loads.</t>

<t>In both fitting functions (as they are a double Primitive Function to a
symmetric function), the "central point" turns out to be equal to the
aforementioned limiting transfer rate, so the fitting function parameter
is named "mrr", the same quantity CSIT Maximum Receive Rate tests are
designed to measure.</t>

<t>Both fitting functions return logarithm of loss rate, to avoid rounding
errors and underflows. Parameters and offered load are not given as
logarithms, as they are not expected to be extreme, and the formulas are
simpler that way.</t>

<t>Both fitting functions have several mathematically equivalent formulas,
each can lead to an overflow or underflow in different places. Overflows
can be eliminated by using different exact formulas for different
argument ranges. Underflows can be avoided by using approximate formulas
in affected argument ranges, such ranges have their own formulas to
compute. At the end, both fitting function implementations contain
multiple "if" branches, discontinuities are a possibility at range
boundaries.</t>

<section anchor="stretch-function" title="Stretch Function">

<t>The original function (before applying logarithm) is Primitive Function
to Logistic Function. The name "stretch" is used for related a function
in context of neural networks with sigmoid activation function.</t>

<t>Formula for stretch fitting function: average loss rate (r) computed from
offered load (b), mrr parameter (m) and spread parameter (a),
given as InputForm of Wolfram language:</t>

<figure><artwork><![CDATA[
r = (a*(1 + E^(m/a))*Log[(E^(b/a) + E^(m/a))/(1 + E^(m/a))])/E^(m/a)
]]></artwork></figure>

</section>
<section anchor="erf-function" title="Erf Function">

<t>The original function is double Primitive Function to Gaussian Function.
The name "erf" comes from error function, the first primitive to
Gaussian.</t>

<t>Formula for erf fitting function: average loss rate (r) computed from
offered load (b), mrr parameter (m) and spread parameter (a),
given as InputForm of Wolfram language:</t>

<figure><artwork><![CDATA[
r = ((a*(E^(-((b - m)^2/a^2)) - E^(-(m^2/a^2))))/Sqrt[Pi] + m*Erfc[m/a]
    + (b - m)*Erfc[(-b + m)/a])/(1 + Erf[m/a])
]]></artwork></figure>

</section>
</section>
<section anchor="prior-distributions" title="Prior Distributions">

<t>The numeric integrator expects all the parameters to be distributed
(independently and) uniformly on an interval (-1, 1).</t>

<t>As both "mrr" and "spread" parameters are positive and not
dimensionless, a transformation is needed. Dimentionality is inherited
from max_rate value.</t>

<t>The "mrr" parameter follows a Lomax Distribution with alpha equal to
one, but shifted so that mrr is always greater than 1 packet per second.</t>

<t>The "stretch" parameter is generated simply as the "mrr" value raised to
a random power between zero and one; thus it follows a Reciprocal
Distribution.</t>

</section>
<section anchor="integrator" title="Integrator">

<t>After few measurements, the posterior distribution of fitting function
arguments gets quite concentrated into a small area. The integrator is
using Monte Carlo with Importance Sampling where the biased distribution
is Bivariate Gaussian distribution, with deliberately larger variance.
If the generated sample falls outside (-1, 1) interval, another sample
is generated.</t>

<t>The the center and the covariance matrix for the biased distribution is
based on the first and second moments of samples seen so far (within the
computation), with the following additional features designed to avoid
hyper-focused distributions.</t>

<t>Each computation starts with the biased distribution inherited from the
previous computation (zero point and unit covariance matrix is used in
the first computation), but the overal weight of the data is set to the
weight of the first sample of the computation. Also, the center is set
to the first sample point. When additional samples come, their weight
(including the importance correction) is compared to the weight of data
seen so far (within the computation). If the new sample is more than one
e-fold more impactful, both weight values (for data so far and for the
new sample) are set to (geometric) average if the two weights. Finally,
the actual sample generator uses covariance matrix scaled up by a
configurable factor (8.0 by default).</t>

<t>This combination showed the best behavior, as the integrator usually
follows two phases. First phase (where inherited biased distribution or
single big sasmples are dominating) is mainly important for locating the
new area the posterior distribution is concentrated at. The second phase
(dominated by whole sample population) is actually relevant for the
critical rate estimation.</t>

</section>
<section anchor="offered-load-selection-1" title="Offered Load Selection">

<t>First two measurements are hardcoded to happen at the middle of rate interval
and at max_rate. Next two measurements follow MRR-like logic,
offered load is decreased so that it would reach target loss ratio
if offered load decrease lead to equal decrease of loss rate.</t>

<t>Basis for offered load for next trial measurements is the integrated average
of current critical rate estimate, averaged over fitting function.</t>

<t>There is one workaround implemented, aimed at reducing the number of consequent
zero loss measurements. The workaround first stores every measurement
result which loss ratio was the targed loss ratio or higher.
Sorted list (called lossy loads) of such results is maintained.</t>

<t>When a sequence of one or more zero loss measurement results is encountered,
a smallest of lossy loads is drained from the list.
If the estimate average is smaller than the drained value,
a weighted average of this estimate and the drained value is used
as the next offered load. The weight of the drained value doubles
with each additional consecutive zero loss results.</t>

<t>This behavior helps the algorithm with convergence speed,
as it does not need so many zero loss result to get near critical load.
Using the smallest (not drained yet) of lossy loads makes it sure
the new offered load is unlikely to result in big loss region.
Draining even if the estimate is large enough helps to discard
early measurements when loss hapened at too low offered load.
Current implementation adds 4 copies of lossy loads and drains 3 of them,
which leads to fairly stable behavior even for somewhat inconsistent SUTs.</t>

</section>
</section>
</section>
<section anchor="iana-considerations" title="IANA Considerations">

<t>No requests of IANA.</t>

</section>
<section anchor="security-considerations" title="Security Considerations">

<t>Benchmarking activities as described in this memo are limited to
technology characterization of a DUT/SUT using controlled stimuli in a
laboratory environment, with dedicated address space and the constraints
specified in the sections above.</t>

<t>The benchmarking network topology will be an independent test setup and
MUST NOT be connected to devices that may forward the test traffic into
a production network or misroute traffic to the test management network.</t>

<t>Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the DUT/SUT.</t>

<t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
benchmarking purposes.  Any implications for network security arising
from the DUT/SUT SHOULD be identical in the lab and in production
networks.</t>

</section>
<section anchor="acknowledgements" title="Acknowledgements">

<t>To be added.</t>

</section>


  </middle>

  <back>

    <references title='Normative References'>

&RFC2544;
&RFC8174;


    </references>

    <references title='Informative References'>

<reference anchor="FDio-CSIT-PLRsearch" target="https://docs.fd.io/csit/rls2001/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.html">
  <front>
    <title>FD.io CSIT Test Methodology - PLRsearch</title>
    <author >
      <organization></organization>
    </author>
    <date year="2020" month="February"/>
  </front>
</reference>
<reference anchor="draft-vpolak-mkonstan-bmwg-mlrsearch" target="https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch">
  <front>
    <title>Multiple Loss Ratio Search for Packet Throughput (MLRsearch)</title>
    <author >
      <organization></organization>
    </author>
    <date year="2020" month="February"/>
  </front>
</reference>


    </references>



  </back>

<!-- ##markdown-source:
H4sIAHjIYl4AA9V9aY/jRnrw9/oVBfmLtCtpZpzdJO8EAdKewzuI58B0zxqI
4V2UpFKrMhSpZZGtln/9+1x1kVTbmwQI4i+e7ibreO6bq9VKda6r7Ev9qW02
ZuMq5zu31T803uvPpnONvrWm3R70vmn1J7P9ajt9d2ib/v5w6js9//TDZ08P
LNQ32mw2rX2AtcIv1a7Z1uYIy+9as+9WD6emMl9Xm+P5fnWqWn5o9fwf1M50
8NC3z799Dj+tnv+jUu7UvtRd2/vu2+fP/9/zb5VprXmpm5NX5/uX+jtbbw9H
03519b3+seH/fw/nOqmv55f6Xd3Ztrbd6jXuq7ame6ldvW+U2jY7ePSl7v3K
+K1z6uReavjvG701NfzWatO25qLnbq9NVemL9QsNdz8Yf9AH21q4p17prtny
P3zTdq3de/npcqQfND7wEl+Gf4ZHXtI2O7s3fdV5eCL8nV/ix5Xpu0PTvlSa
/lvJ/zUcH554v9b/3tS+M3V3qZuz2/4S/85wfm+2zn69+lDTwtVfOb8FvF58
Z48+/qltkAzsznVNG39pj8ZVL/XxK6/3b1t8db1tjtPH+/Naf0IMDw7159Z0
X5vBn/6LR3loiYayk6i6aY9Aqg8Wgfb57atv//iHP8g///nFP8E/FaI+PgN/
efvaNatXt+/uVpFUGeCdae8t0Mqh607+5bNnQL9+vd+tXfNs6133rK08kOOL
Z609AVKfubprm12/BT6pnx0tIG7XVM395a9Az+avp8rU9q9d5JbiiUj+60N3
rHhz5sTZ29ewn8bj6TvrO/0+vQYkFk88o5dyzvkW71ZwWkAcs9wx7Dl92a5p
Kr92ttuvATnP8FzPfstqxeHfA227U2V/uwh5H0XI5I1WqxUIFt+1ZtspdXdw
XgNW+qOtO212u9Z6b73eHoBZbX0P/zwfHGxvTqfqgkIhAd0Be+2s37ZuY3dA
sPonIZafkRc3uUTxzb47g8TRG+Ph2Q9v/6znH2x3BkGj3vY1IVz/2bVdbyr3
C16yXuDBjSake9082FaDQLGPna13sMTJtq7Z6WYPkDrapfINHAz+5TXwvm1b
eAQOYbye+cZ81R0gHg4yWweIJTLSAnW4YdsYgCv879TgMTcXeKyAj/c9btEd
DLxGXKZ7OE5L62vn1akQ+wCT2nR9a5dw9p2umw7EFUjSo6vpgTWj4+h2u8oq
EHnvG+Apur5SAh88zoODPbxGADqUxLAZQzy7hdE1v0Dw5rOBCFYeRL5xNWyV
0BPuUSFNtURTcM17YOcaL23KU+p9wFCzV80eoAubV43ZrfV3sDEiM+KXN+aj
jq6rbY2nXTPVHc1XOEL5gKnum9Z1B1hh7nvABSDwO1eb9hKIHvCe6EwhVH/6
LTz1sz7DstoDcICYu9aZaoEE0lpAT60AoqhDlkjssMn5AIAAmWQNwtkfmjP8
VJG8qy764O5hJdhjZ9odXODBEcaWqjv0dC3EgOtAxbVEGg1QB2i+MxLNrDvY
DGszDVwINA+rOg8o+hGgZmFLA7A7IMUHeISTdQQ0okpXJ9gdkS6A3Lfbpq+7
NZLSZzowIO2uCUIc2R042R2JbQrKdp55FhSAjXzmGecZX/vedWYDtIqyp+Dw
nPgApievA/4meF+ohM55B1fw+gYQeYNUecIzg9p+3VuC3MEWF0UiYZbCg21t
i7Sdr8m8idvBq4pFcse8HCV9Dlb76EgyhAuUt8JtPZpK8NypcluiByQ64Lb+
uAEMwSlqe+YH+S/pFoFhSafZHV73G/0a6GVr9RcSG6iPlHpXJyAlOC71bEfP
zoDGgB4A70b7k926PQDh5OyWYBDehLv6r3A6ovMTCzkQHVuQ57DYWt8iOnhB
xLbvWyBCEqX0BtGbiscAOjg1NVBGxogN8J7pMvHyFW3CarHW74iA4IwKBKd3
m4pw1/a1bOhpjwalVOXpDzVBF5DLdJ62I0mpkHXoHMCYTd/iCiRd4PeIS4CR
RVEFLwNQ3wItnvoWZbYnymZZv6T1zwcwfpAk8S9xWURUBFx2V7jFFhXfDtTG
ULzP9Pz2yx3c9sZr+AexDGxwMi2urdJmpAuICVBakaVUM66Yh9Ew3lhQpAaQ
wKSZJBqrV78c8GePd6NdQbpZsyvZ8vWXO4CDUFamkObwh0XYr7WrRIss4kxt
qotnQofHRQqqnn4Fh+3spnmEv20Ptftbb+FUm75jDoPDAT49KCaCAhiPJzsS
K0zxbI4WFH870p0EW1zVBIjqSYjCL7xVOVyD6EqcC6ymUdIhYisQHZnNcoFf
1igzvGLKw81I7sOyerCsdSKEwVXaXTQcAmSf8we7W6ILg/RuQB1uu0DMhexg
aAoIAHOvmnrv7vtW1PsXT3J/OWFHwO+bM0DRka4F7G/zV5FFwq7KAdFmp+Z7
o30wFvFB6RvRgqpYFR0yOD8QxRZBUadz3yLcAWXu6CrTgqYCaEfSW6KiQ6zR
4jumDSKI0a1gA0Ub9KcdyVGSO+gTMhuRxIKLAE/uSIMGJwMOB5ugXkQCtPBb
S9xGWjyw0VFklUUsAPgWqKFbAX8wpe7AUgDhmWyrjn/BdNddmITJxDJs8QRR
M4EkEFNdAex46XlY9t7WKDKbdrFEDiaT9N4QUWW3U2bT9NegViA3aTA88Ik0
Egrbpia0kE+PBrtNiiReEbDtQdWRhGRgewYOG8RAkLXrrl1/dCGSoJMnBusO
dKGe8R4zPCldnSS/QR0Z3Q8xL94gQGpEfgvuLZ8qHEt/ZHtTKflZhJmYocEs
AiIAdcuk6JHY921zHJ9aiUUxcWj1Bk3/YN0KNeaEHa2jFhQvGII7FAFgQHdo
hxs0EttOoQdSnv+zPD28QFilvAGT9RDW4PuAzrd4QRXZVHQ7ybrLyW1ZmJBY
Fzj0W3hg36NxmcEErKIxlSFchtvCPd6NedsimCIIBE5JdpheHCdTB2AqfggE
HQM/GU5Chmk5gA06DQCeCg23Fres6cjjl2T1EtrgJHdDSCOOSigTNFCT4WZx
c0IjIrC6KOA3gMITECDE/8/cnpa6enN5QU3ePOqQJ1ZGkwb0YE+Wq59+UkV2
AiO7v/JQhJSAXH8kQfZJllCFcMLLkMQBEkNhHpQtAWxjD+bBNeDRfq2bc02E
qTIHYr7rg7G9hF0bdKPwn/cg3jqg5h24Nhe7A6FKiAnaFylZBQOZ9QYcuYJn
mHs60vFpcyD6PbJKLVJh71pPgQFrMcJp0UcQcRQ0B4Z19w599QL0AzNvWpLv
wJ+sd4A+pLMZKpsZ2o1s77htD4gPAJqje2UfDZovoK1/sQsOIMzIVPUO7waS
lXcSwKkpQW8AoY/BW0r2hGyDZ8D73YBoRgI9Jv89O7cYTPoYIlEzjNX5WaQL
5rMYGUA/FM4a+QqxS5tU7BL+DWM8HSGkAecM6WIemNHV26on7Y9BGPDunJj0
qOpVruoX5BCIsR6ufmLsrAfWyhgXhekztLWDoYRaVYFjMFx+aCpRIK5Ao6gU
ZJJAON9HHaTuRgK+dCUmDd8loA/dmu1Tchy5xZ8Q/2gVoHtEZyPMEFAFYyTq
MIwBFzwBztyWvHrNYg+sqGuyKuDYCesKXcImiiIPbPcWykF2g222DtiPQ0D6
jCkBQA1f6r4Hq5rElQov8SUBBi2cLLJ8WHOxngLjsSdTbI9nRosoEihwDWAa
b0g2Z6ZlleCU70Fv0UWA6wIBZW+HBU0hDF+14CgBgWIqwtXiOo21JLuw2Wr4
Y2X3HQVTGNOF8ELwwM87tiiCJQQ6zoA18TGLw5GKAiDfg4AE89JiSM/qeXQw
AQHhCBg+82DH1LsFCp+hkZjx8KQQY3FHyyMJ2FOHXgSlRojS8K+74FEEvs1N
9CBN8dfv068Rm4MnxRdkK4cCHV3LsQw9B2ICEd57smyAORZyUpWxOGjhSPfp
eiWf5sHMJbOxyv4cboLSa9+xUhYQCGRQEojtR6ZDZia6HVIVB88pFDUiV0au
1xheBM5L9HpG+x25YbBxiAqOoAoQ4bjLhQk8SfpghRCZIImHO8WwgDuCZeMA
o8gwrMfIphzZznDLvgJiuq+b1o6JOxk/I2GBxiojTI19tpwiXkfvOPyLkCKh
z0nb+Gz8WL6ZjCyLgDXF2mChpgXNvUUGRAxy+ABDq8R0tsbDcXQq5zJyeUgR
RRIX4Uk+E1ptY4kJ69EiS/GY2D2eoAawbcRS35FvB/qvRbRMrIk4drUSkQov
iVBFCkwYXut3eED0s7ZAiaR7g0nGp981FMPYgql5bzPnkiMXJLeBKj1gnIQS
ABPEdu40UlIq2tuUTSBRdEElD9qgQ8FNCSLEjgkLXxEM73KT3bMcXsYchRV/
SxIWMZETdv3Fto2eA53WTb3CHxZoIEkIR5Qq2XbhbE6CiuFFHV5cYjoAJTFG
+xdi7VL67RUeSakP05b2nHia9VTXNKCOtgcwUzA1M+bYzUVNm4km5URydsr8
siA4kbOCvsxP+RmApdQPEW4DDyABF+QEZpY40UUnTLQT9lCTymOwnWvSfpxL
+vUNrzoh2f1IuiJj6LlLtMy71JjBkUVCmAKxp9DgAaNtHm3JRS5iPlJO5l3N
1jhigmORQQORs0Eph/B3QRGpowmFiOk52BqR/59gfyjxFSm7dOJo6wT6JdTK
AgGMoBOQjTxGJ1DpBFvU/3f4HHuvHLkFqDmCHrK0/VuPi04ExJDAo1RfEsnF
FGVHxnGz8bZ9sNmKMccQDinmqTdHtJP8U6twROzBtp5UAy8Y1jknVwljl5PP
ZtcZaIksfRPTsFMMhKedRiWL/yQSj8i1ExwouUt4w7UoXqfx0nE03IXch4bj
Xthoe0BLFy2ykGdC5cJuqmGHU2Wpugl7SH9mSfkazGOQnj0rxR8Z81dulxGq
7zf/abcdG3Ixjzl2ogLASVFGpUaWcGEZ/WDO+OgPGJrQLP28cuCdOsGCoH+X
8/q+JTxuHaV/wPuDnQDZ92QkpUyWyilpl92Xc/4xoZRW9mupW7j6ZkrjXOM8
lT/PutmVLjocOPNB9+4xJE2ile8w1skIRUlzjSAxRO6DqzhyPK2SvwyczOUV
+yPGvXPjtNj5Ov0vfxs4wC6wXrL3+KamQAqzMFDB3p5hUdCfsNKYpJZkZNeX
8uggHOcItc4u8oMD6d8Aks29LVTJpycpguIUcTXJ7AEJY4LU1ujHesXqJ7oh
GaGszhb8XtJMsrOcN1os9sFUPTEM8BzBldeeB5I6UDaY+QufjTGfsAQqnHCv
ssTCI9NcriyRPcquCBdkXMNZARrJLQcWntJv8Y/XlNvTckVyZGMLfuo4mrkz
V4w6U4wqaDyRxbBgru9C9L60SG7BdGPeC7Uo5RMENFh423suXKiz5VHLcGAT
nBf61ZqrIcjlR0nlkttJ0B/jj92EHDKjM5QOBkn7M3lNtaVEPFISB5B/Gyfy
lkaD/dwcaaMc53qgzVpbvM3an5KJFKf4OxToboJWluhWVXBgcfWLi3sMJnIS
Io+KjS0SxaJjWJLB9IuhDVB991wC8pbCalNHiST5m4hRJbEOp2vqkUsoQXLQ
TrUjO6qhQkuWNJxTA/YsBJoHo9WSdS76SQhB5KTYx/+BrgVLNroUiIVr4QcR
AygjjJN0U9WgydCwh8KESMsslYR+cwWIQtBaqmQRP4ZfuQoi55V9BOIBmJDX
g0Ia6JUxHDkHK5PqLEpHcULiT3xJ7vl9b1oQfZaiU//d296nxfACSu6sn7zz
r99XUVAdYVPb+8rdk1HBlso4wEmVCBPQUf8V6HyAPV8XFUz/LfjgFYqCqP89
CGVwUJNQoBA07czhVRtB0h7hDAwH9snQI1A3FZYJ3R9yQS36kn13vmrIOQBz
bhwYTi3FAaVoRk2Ua9pLKp4E0FLBhw2+f8ZZqTqEEADgP6LSl3TcCPKBHyVC
hQWZffsriwzIW5YItWH54kFx3+jD5YSRP1ZlTwhEjv7VDFy5USPAXSqucbl2
Cab4DikCIF6btm3OoLwpjg70YclDQNtdzZJazZDumOJmhOmt8VY81cLWiU6/
msgooMVJhjgH08iKevq2DHU/UWmaPAC8ryr/yK+L60W1mCcsxkSDIJbTis+L
SRLryygiWTaUlcCYZ+StU4pHqWndR0m46GUFUfuPzyWo4qWW7Ykr+ycxzJHF
q3dgmcLBQ/rDkPoniZCLo9UNmMiZO0tFlsJeZM9MRe5JaeZ+DIehWZS5ad0m
kbhukaGwlHbRYODa9WQRRjcsqyLmmlP4Y0yd2gIlM8VEJGGJlOQcY48zpFwL
vJK0MlbEh+JhMAUcZ8BMd5XL4NhlPTOnbkKuFwmkJNVw2VuuXMaaSspiTD2m
K9PX2wP6Nvcof7rhcyqZ+5kYojx48C0T9XB5Vs3sz0X+EnImNHG5Qtf9mnCE
dXbdAZk5tjLAJfzfd4urTEyKgIu4gb5bZ3cp2sSEfk1dcqkc6fiBkzmNg7iu
WJOqYSZKTIH5D4mOF10A0yxU0F0Wy6BXMv810bCKUZbMG5+iUxSivHz05JlX
iW/VUxogt2zMxpPZnxixOGRixJvB6cdV1lJWT65QdpQpvD59lnghSZfUTMCh
IPQJMFIFjgv+WKh0vyZbfqS83eS1qDYDfnE0UveZy9vgDmDQRwXXD0mZsk37
HlDV7GxFi2yw/hpFQR6+LmQh1hheNzokaFXsUjVkrA20jjIVxzp7imdmpiCG
B1F83zfNTkfuDt5Y7DgKPTrilAEhP+2WIV+MHejcD1MDP4xrXkZVFJLfXRf5
hdiPIgqOnDXKPoYcdHziaE4nqj/KFRDGH0enCwolX1zCXACyvqZ6B7EZqKug
CFm9q5EA4PBwlI7TGsm3JUkzFVBAzdh3+CbgKHPP3ZFQA4hx9kHS/1hlQDYX
5/9vymQ/tSEUufyS6So7vrfiUwQ3B+gZ9kOK5uulg5JnwUWcfhSWJFvH1RyP
4zz2lax/cQP9hlnIslyU0Bm12EjO8BSLCNCnCWHgbViEMJnXnCkjfQcYNtOG
8uYsMfFScj5zDP7MtUz0W9cRXyeSx/NFqkg7Ukk+eRS1OIKxMh9PLBSTgVGF
RbKKzRgsQW+J/IQQ0URxBmyANc0Hc8KuWVLIe3tWkcyISkB2mK9Sro1P4stX
LsFLxoUo6BNTjWnVkOINxHyLwQ4U877fyOk+Y1LuVdNgqTbSHz2y1jdsBLG/
GF7nWAkaOuoB3IBGckKwALsDfh3PmzMfWXYcJiII6LkJN1DS5BIOvCBZeHXr
OecaqH1RXDIRFBkoKUuob6Q0Wd8E5UWw+BO2fFXAi1VOnEVIbbu1p85zzMiR
NAA26kUjd1yWS4n3h6zSwk8lZQya6u0lE1tE7EAJeQGHl9orNn54YyV6iYvm
ClYp4n+D8xFYxcG+x55v8kSoGpJ+DGUubIKG114G5mlRwDlO/ZOlsAFPSM0D
LhcCm+F9inKU/C5SzxmAkieV0BAmX0kOltc60prUpoeCKt5NYrusG8KJ0WQg
YbltWlB6HRVyRp5ETYv+FR7paB7p30Fx0A2A0LFkUo0kJbW5o3mCnUZb0PQ7
6WmJpwgHY3XTTl2EColGF8mFRl5mpaLO4/LVoVhb5qp6Kx4aZfKRmbPEsSpe
omRT02GOB4+DmwRtdosknhETei74khGNe1HY9h9MJDRjtls4wvaScax0AthH
R8WYu+DBJRGKe4s9pqWsnntbkEQAMFkVh1QmhV5kgUaNZQPUpujBmwycgZq2
ZAz9LlwAz4X+BMlktnIyTcRNv72jrL11Ld6y9TndSCuMqFjqMRt3iXIZy5na
mXeqPyEqgiXIgvc9Vtu+2wE2+MoBCC3XzCZfdSA0CubaVngHruMrL4w9NFSs
NVmQMFgHZANYYrHitNS+pGgrLM/Le3W4LIqKPatma3ItHl5XDPgbrx+45xtr
mABsE6ZDiFabivXdwRaozFeV25XHR0kqZstFxxbv8WGybFDMBE5ZCfE8ReNX
5mgol1e8rPUHMLXhsDWQP1JjM+z+IQHBtWeDHt6BqfGduVjvDCb3qLgPcz6x
zIbjUKBPbSvptM7ei2hVeeswR5GlVBGFFwgaR5aTFJGFDAYarS01WyFd08vU
ZM8vcxsW3Z08CywHLEs2AqCoDctnLWuZ3ClRRYHDgEuyaqM6W1P6MkCbobLM
ZBDo+TPoQLRpQl+M1NYEZomgUaVTSDcwLvbTBG837JVpjLjWVNY7NrgOKuao
IMynRqqDPbI/EPW4CsVp09U7JLQNoRcOZbEkGYtwOKqXVRxeTcwydRHi5Rp8
UwnFl5WXPh8TcORpBJIo7H3Ilo8il0nFbFg5OZBe4V7jBKMH+YdZV2kDbsj/
yVp9Y8fx0R6b9qI2PdIJWWifYug5Kyqq5da42OQ0AurgJ3IIbfyh6DGyeLwB
2jpZ/sMS3XJJyMiDWxIxmTYU2vDaWtZ2o0zjj6EaeEIxDCoAeu4voaOLVRmP
Msw6L1OGA+0OPAtFGVBV63HW9/bLnbqe40UmPDTUSz1M8ZJ0jNEo4ahc1OSJ
9EhfY2d5fvfx9ceX+gdXf8Wzng8XfUaw4M8v7OqfSIuSzWhi5Q/76Av2TOKV
vutdRe0i34Gq+QrE+yfM6pylUIDKuOjkoTQ5vckyMZR6Zb2UYBmyjJYHY0N2
jW3mZUBo3CwDOOdCzUquP5baI8FupLooOH5DX4j9vyUNuyg1gQtrSgPf9is6
ytiMwKUQKQUOy1CcZ7T4wFdCAzs0IcT2JhKryXQrSnqHCyqJrLFmwFAXr6u/
p2IWzAvg4aiSZ/DMcqrCOk7mMFeullebLmNLEXuImY2YHXpkM0Q9g2Em9xUU
4qHh2TJFPCqrbcsrXFn1fmqc91g7VUB9PxIaNo1ZYTN1gI+56bIhD+LRqgGW
FpQQ2BHt2pc4IYWrQ14P4GJChH5PZnYSVXLasthkDvRTg2S2u/tYq/hbWhaV
qKJBy+JU0jgMELCxRQFpl4KCwHfUFgteNpXt1xRS4WEreY5Kf7kKb7aruT4V
vBJ044ex0IkVkcZ70Ke2OgXPM0t0NXnZIram+wHt7A1SpXD7u+Dix6D/nWjD
sne7ayQaMJ1oYpIKbK4KNhe649Q6kXLfUqtHZhZPkHhoV6/t48BTQG3Bu4Nm
j16WuKvJoixSEig3d1hl7ViCVBUwcdZbNbSPQnA1dTmnVoDcYpnVM9zIBv9w
bnbsGmKxN0UtQgt3XuEXDInRYcqlf/9iFrowrOK4q4/AypbDy8Vocgy9h3kl
Oeiw722wx7czuCt3xvotGHvSZJDyAsZJO7kYdnbAMkWx1HyDPmg0AMnvTC08
ZYj3AKIGvah6bBoWLIApRDHf844MU0DLRFujMGC+mQiQopfa0DQntmwo5Ia8
X1TulpF7T9VqpLJ4ntAokybWBqqeKwqR0EPzL6gJx3Ww7DrWwg5SUDw+grp7
pLyzzBPQdAJ0dc3ZXFTwd4eu5aTEJrhzoR7iqampNIDajC1Hx9m15cbdh5KE
lqnnj6LBLFPlf9JdRCmg+HZWZhATF4iWvHr8HQk/pT6LjZzqKUt/figjsO0M
9R+VsTMJjD3sJGhqTEBQPHMZMgND6PjQZCSJOoQS5ZrIFX+UCBG929oUR+za
nt2yiqLWV2vCDEn0Ud6MuGBjdgkPYztFgFu0p/D5ZpItw2VnlDaUEqqCRXh8
Shu7z2mZ2BlMR0LD2u7olxzaZ6WEzqzzWxlz4nmmhFg5cNeZ708naXWcCcuL
NuOU/G51aLZhUJCeS0/ggcZtXlWNWWgtFJNi5EEhnaGwI1iS3xUWRrVMyiFw
NtpsnKBg27WmeTiAtR+torwiRv8aP1a6G4Ie6Ys5ooYjPguN3uwEm5Ghi5tx
qTZhzbQrYJvBsJd8LeoTtFW1iow3pMQr4ku/aizmyxytWTZc3HE4BmxUJPei
BCe1WMpAgb4lI3UkHqjM8hS7MUhaoFQL7lMeE40xJRRrkvRgLhgMGuJIQMlE
Wqp6UgiFmGEhKl3ckrFkDHNacT4dSqo84iC3iZmrMCUgGKiJzckiiMMgMeKV
xmf0J9J3KcxKdgwHWccAo1QOmvlqWJZBwj5lqX7DrCP0eTLjWabmibOLZQXo
UtUOozpV1mtgdzJwEu2CI8b0pHBwECymMXMNjs2kSgbBCmi12sf4no8DJlw7
Jkqp2XSt+EUuG/skBPsGY+SvQoxcqTdlzDyE/5NxKT2P18MM40huFOnkvZFY
IoElMUEMXEpD/IJkFIuCPDyf3qQIY5j0yBMHI61JpJ7HvbIJ+3S4PCj5wXmj
Kxjs7hSD4C2nfUW2Ga77wGluJXg1EtdAC6bw6u+mzAtYvnJeImV7lNXsPo1i
chyFpWQdDS7luTCo8hCIUtYarc+pGMucpXwcxcjyehnnLb5v4GH9yrRVs8iq
CODvQcwNCj2+0R9PsLvMXfWh5zw0FZPZsAxZ+CkPnLq7NmhE1xgFbg1PJ1Um
lJFiDX3M5A8CDsTF+f34PoFUEF7K7NAnlPAjAhbnNT3YJPLyjXGzpUjJ6MsL
YMGrQUGOtB0m8qh87xGlmofG7TjAswfvUlu4D5AJ5qRREJI/s1T5uBbqpQUb
LdkF94aXFACkUwXoZ4MdwG+s7DbRMTUM0Wy+vgqj7jhOVLoh7NcNyO0IVorF
cSthnkNwFQPPzYkDQ58doVYH5TuSVTrUIUz6l1kIBWQV12SjlJ8WM5m7tQTP
u2+H418zfVf2XQTwiF2ZxmuShSNl72xCIM7Rq2eFHdx4dKNBf8UxRnct2jk3
PAkRiP/dXvxTFFGpTQxFB3bSUJ8UvgGK+bKxMe5PZVR4XPScK7dpXX9cwDme
lm6vPn754TVPLA44yt5P+Q8V61jZv4YDLOn6pFdlgEPGnlzlgMWmON8YBQP+
bZXmgbKi8WD24vFgsb6mRB5ovAp1GHsyMbfMVw5TX9b6Lfd2Ut6cq8NlGCnJ
oPS4oVyIojGlR1tURWWDUsVABCOKc97zEPy+1hCtp6ukOTDGNsfnN68+vn//
5sPrN6+lCRY1Dlcn4eE4NJjNWpFRRIhPMDywKOAK8ZLgoNwvUdiuNWdR4Vju
g6Ko6lm23lRUHBRcJN5YZm7ioOALN3eFsT9lMtzjGAKMwgSdGkPS4voGc1R6
6XjOieUy8m/0v1Px0bsiN++zJCcXJ4WJW4McfoEnMpowWN8/qreUqKZnsqnq
DYB/xckbjJBj67H+aWIm/M9SYZXeLM+nQ+i5THzLqBGPdRl/pZiZFGbwD0zr
S65uQ2ldRFBG+WzkGvSwsL1ERh9MDRYRRZ+M1h0F+L20NqDBvHFMJxSeCitI
sV+sHvKwVqiR57P7wYs8rCcVki0HZcVY2XZfNyQgsVSbZoNS7UxYxNMUTh5e
ZHA3HigZ/DVums6bLqibg+zsLKArwdmd3fT3oSzxyhjFikpVwHDO7xFlzjAa
8RpToNQYaXCGACrJANbouGQ0IRS05JgAkh8V2MLlnq//qELCEdfMxlNSaIP7
U8glAKOpI/P6GEe5h6FnMvgpm77F/s/8bvXZPi64n7qMmEkYhef6gMa6Ygfl
VQBs53GUCy27OZXc0O2xcCvrc+I7mjSbhduuuCufxhGBroENJRTiUI1FR3Dk
UCziPLcus6vIQMAAGPhbOGghlUHykbJKYFZ+1J/2hMnHDZOZ0UWzimhwANJb
6+2StWywFrKTZF32DDEa3UMlMeKcvPr0xY+4QIf6TkCWzJPjiX5ou9LcMdYe
IU1Kza/yQQMeykgrhaYylbm5w0RnMBi4VZ/xkQUJ5mJHAUb+uH4hJOn1RCHW
khl/WGolCu/5+oUk0BUL+DBs5jOKWXziDZubyKpfgg3qiZWaMFueZR7WFSwL
G28g0skm9Tw7nWpeglU6H9StArzvqXYqdrNzP3TeIh8ar1RNAYMQB1yDCZsZ
uxwD8zJTUkqEL9w5whKYBbD6dAFEYjgbQ22oAoI00bMPTW3TDFk56joqisyR
xli8jJzowqhy/H0sJpgVljjYHqxzefQ3yo8ZzRRUoe6UgjX1DpCPh0iVgtfD
4K+mQU/R7O7cjJmVOrjFs1xKiCCVcCLYy3oYDg2EsD8QN1tYPCaS9S9IeTYM
DH1wAelnSXZjc5KgfGJnAJLnHGvERcana05jnJ23pUXsOzBgiwLTolgH6aKv
sb6XjHoy5IBs0RilgYRFyAhpQVJ9gRtZDVFAnmz2R5pkF6tCQ7l7TJ+gZMCH
0woiwz+1IBgoZh5jfTSbMgvF5/XGps6WyCJeyHzXl8qDZdlifPK4yr/IMMZR
+IdGvZGwb/utzEvaNT2KnLinKvacODwFQnXSKFyVHryCiUtRlAh+xRIkhO0E
bNyGeRk0QoRYiEQAjzwDTnCcLqTmVjg4zKLjWVoSW+fRq1ThjCkRDjIV9hnz
oIzTcRiiezCuojhzIKARECmmavSM1VHFafOZfKdh5k84vn22DP1hIHBkDIc/
uD037KF3BpRKwwXBHHhcmUfnJ0awRHtnQbS4zxCzsYy4xEI4yupWVoVFL7So
nsdkAT6ishVdGDMjNmk+lYLqlDCsCCgjDgDrgEx+rghL4+5gIwItZbkLyPJs
iit0OI8lkpQWHJJgSfYgYcDnRBsrrrBgYA1RwKXpNN2cIJRHI8DFb/h28D4e
kvbiaY3ponFw8vWoHRYzYQvQTs+ObTuTAY3YExQVFhmW77GUG8S+jOSmOWli
1LPcR1/V5gnx6yQnLYeFSolIzXyRVjS5skmTx2iSX6cmCz8auxTlIxeMGJ+U
tl/qHF/ELVmOAQHNndwx353aSfCqrDJ4rKTM9HuKtbwlTwI7hA6sWLhiF8gT
1LTl2Z+0+lKRvYN+cew44rE9FD0D1k+hNFfnE4krswXnQX98CEaOROwskkUd
5oVw6DO9xu2b8W48AUb+qELpPOAEP2S1zkyoOD8AkZQvnTIqCWQ4Ao/DUVk9
viwqYVf+QVLBMRYQz9U1UnWKLTJc/0IRnElmHJgPPowTUGkAs9vP9Ab2xFjA
kuM42Hncp6GGRtLvPAXCyHGVFJqHsdbYMgR00sENUqfQXW4EJh0TSgbCd8Ai
MZLgGksKLNQAc5D1Q/gle8fUrzfzvHOy7vY0BVPaHlKi1vFABa5bAXmH9mto
dvPhs073R+Q2rA19kLxB2JFaFqntiYImct0h0F9OFGrN20Wq8qIZ5WXD4gaE
HgicLKI9P3InEquc/A9msVSBjzU1C+Kp8EI/NhV+90BXgKAeDkAf1NO61f8K
b/1u/kL/Xr/5y/z4zCwWvwN4/jSHnzbwU/b7Z8VTPy+eyT8FxW/a/a+i143N
jkLmf29wqhIwTcSkSpgEpgplOxQwIFmnkwWVnKJTXBxYIqw5QBGs9n8EPYgf
APVqPt/olT4u/vLtM/OXbxcL+IF+fQy/ABTd/q3tfvrkfgZEHX8HGNn+BBj6
OX6N8fda1uC/zVcbfHABjwTstnt6YyHfOyAbPs9WS8xsmDGizMSJHeL4qZW8
fRDtlpT3VPNRgnSRZUgpPRvHqOj56sVSvxC7kYQZaWCeVS921zC7Fs1X+SIe
WEBH/tQYBqCXFD3O86c038PaHVpTr52YC4bEGhmm4KI4PDhRXhkUlBgenylh
mwP56Dr90MALZdWkfJTmdDDRWMEZ8OyUkMGIHlDDqhPJi9yPM9b40+cJQinG
i/gNrnxoKh0nCr4iMS3xKFydrVOpqOHTszvcGsdfkVEGBfoObsxxk6KDXEbt
/AvXNbouuzCYPg6TAKZS+a0HWVQcUs85QCxUKEsTn4gJTdQuq9SHd49xRs6/
DfKEPA6SXAHM3ZUZQcqJyxepsswm4+ndEV1S+uzCLcaxePa6lcaUjaOkSVGR
6/B7gugIIJFEoVYOg+T6DjA8NoQQdF/5mxz0Hs1jlSRRhjOOa+7hEunDWMIe
kV/SBG9+XOVoF+qgEA4XSQTTbduEjdECa91jjDRN3BChVSSLWPCS2OPQ6bGJ
w2hC8M/TJ464AG6eBsTkrTKLZarvZHoicymWZwKp0Gf5qNI62tFkX6nDBZhg
tW+2/fC0aIbQh3fy+kvpGIrbTd4y8H3qTQszxou15jyah+qopdijmwBoCjCp
BLPy9qFwmaPqmmuTQtCMPlVK3x7qgndTPsAr+jjlZVBzKg2lOfZ5tVDuWrxP
11lrHgebMBCwiZpYoklyTJWGIXN8IPGNhLWCA5oXieOT6RZ4RXWFUApQrfW7
8GGSc4z3h1A4ikaQTcoCOVQ7/iUXt+37Sgxi2VM6tWkuBME3VGjWcc6DSlvw
90AE/vN727CDuoiGgwxHwzgcb0Bt55xx4fJT+hJnOHFKD1D4bkwyGC/A6v8T
hZZVES3GAhAcefrP6+eaKnjpE9WLUAUN0No4yX1gf48MFd3QZwskRhIrKjNB
KGlRFeQ53gUUlbd0FbKv8CfADGcVIo9McRBO9OeG7o27h1t7Jh6u7CaXCyNp
hDmuwwxEw+MfubVTYnKIBqrx+PsyBpqTfCSV6ORqLluzSybfbgxEf+qr1DEZ
v5saOwYCTUxGJbNalyvVFgxABOmoBS82c3NM+HSyNL4LL8uf7qX2NY61saSn
+gM0EMQeWesPVJ8xXF1qCt5//rzCshB0qtx2WRqyNAqBY53J8ohzXyRpM2xE
xwmRZeRKloh+OVs38dd5/ALDAcbLINTfVGlSpynhKfoVWA/T/yErMR0yXoZH
pe5uaEawbuREGcbi0fUzFFrJE7Shtp7a/Hb9Nki7VFufhourVDNc9CkQSWbL
i9yVJC5F1vIaYCmf53h8Vud+FuYlvOzyvwAAuTx/rW45nI4h1DhcGJ+8hEho
KKjI5rAjL2IwwHI7IXVU8ZW4AAOhg3MOmmK44VRTKIaAa0oN08e1lJhgUk6Z
nYMIsKU9U+E4HjqaQTHyH2XtYOYhqUhZgsQ6bjc1iJkq4dJyYgAVrwZdrcLH
zIYNLYLDUj8XK7Cn67lHjPgn06JEI9ueXJS8sDzMdSUJHiPZWF80TFVOlxst
8cA4YzxMsKKKeN/wyJzhTqE7mwpKB4M8voRekIQxyqmFS15stxjikD+27XgE
lAraeShn+jqV/6ePgKB+KKYEvsaNqNaJavAGRID97zQoXgoIBUT0rScselcU
sS5lBzXxSgU9SFdmYvyGBkUNix6UK0k2Sob9AcB+kk+G5benagk8tNf/ICRx
DG3xKA/5W3DG4bm4qD1hmO5IoSMwKc4y7p2ndeAxbr/cccHNu5sPN1imm76v
Ah75h4ai+FYme+Az9PAtUFhLgenBC9/lX5ClcJbE9HxWXU1Gl+P2aGnaODqO
/Sr6WK986Rb0FiwBuviXWNZj8FvBz/CzruxSYWStbUj0IP76ytFnX1RlNg0Z
HfhxmQfXNjWPBRHfaBc+hc35dWn6SQ4L5rNQUvnwPb5watL3kv7bNA/BRZ/6
lDimLPkiob7UlNPGsq8CY3fb+y+3d/rDxzupRK1jODx8g5pddnMJH5llCU2f
vwyfgaUvM8hnOwhk4SwoU50HndClWiExkWkFYGIQYSRg5RWMavUtenvL8nrO
py9yyMcAN5XZfl1tmscZtldj5gn7KWhYAlhA0llRjHSjuVY8tuGRC4fCcQS/
oZEXJZo5cV0kEtLtn6jkEOFEnz4PaAlkUbQjYU69OHv4zPZaaxwoRZ+A2JpU
Wh3A5QN9g9HMmcegOMI+co4NdUfWYQY86xazkTmwGSbCp8qY2W5ip6qUeWGr
xobqPUgzrlYrTVOI/j/UZn6OkocAAA==

-->

</rfc>

