SMTP Service Extension for Greylisting
OperationsSantronics Software, Inc.15600 SW 158 ST Suite #306Homestead, FloridaFL33033United States of Americahsantos@santronics.comhttp://www.santronics.compuremagic.comeharris@puremagic.comGREYLIST is a SMTP extension to formalize the widely supported
Greylisting mail filtering method and to help support SMTP rejected
transports by following a new formal structured 4yz server temporary
rejection response by including a "retry=time-delay" tag string which
SMTP clients can use to optimize the rescheduling of the mail delivery
attempts. With adoption, network overhead reduction in wasteful mail
delivery attempts will be realized.In 2003, a non-IETF technology called GreyListing was invented by
Evan Harris as a very effective method of
enhancing the abilities of SMTP mail
systems to limit the amount of unwanted, abusive mail that they receive
and deliver to their users. Mail systems supporting GreyListing has
grown over the years to become a "pseudo standard" among many SMTP
operations.This specification provides a formal IETF specification to the
Greylisting framework, learned practices and introduces a SMTP extension
to reduce the network, traffic overheads and mail delivery delays
associated with SMTP Greylisting operations.Greylisting was originally tested on a few small scale mail hosts
(less than 100 users, though with a fairly diverse set of senders from
all over the world, and volumes over 10,000 email attempts a day).
Currently, Greylisting is in use on many mail servers, including ones
processing several millions of messages per day. It was designed to be
scalable and marginal impact to both administrators and users, and
should be acceptable for use on a wide range of systems. Of course,
performance issues are very dependent on implementation details.How does Greylisting work?Greylisting works by leveraging the standard SMTP client design
expectation to retry sending mail after an initial 4yz temporary
rejection response is issued by the server. When the greylisted
recorded SMTP client reschedules and retries the same transaction, the
GreyListing server will allow the greylisted recorded sender to
continue with the transaction.While the idea of using an intentional 4yz rejection to force SMTP
clients to retry sending mail would naturally be considered a radical
concept for the IETF purist and most likely would not have been
endorsed as an IETF standard protocol, the proof of concept has long
been established as a very effective means to control certain types of
malicious and abusive mail senders and today, Greylisting is a widely
recognized mail filtering method and Greylisting SMTP Servers are
widely implemented by many in the IETF mail community.What sort of mail senders does Greylisting address?By leveraging the SMTP retry expectation for clients, Greylisting
is very effective against mail senders who anonymously and randomly
perform a "Single Shot" mail sending attempt and will never repeat the
same transaction after the sender has been initially rejected. The
high payoff has been the nearly 100% of all mail senders behaving as
"single shot" mail senders are abusive and/or malicious in nature.Greylisting can not address abusive mail senders using compliant
SMTP mail clients. However, it has been observed that many abusive
mail senders will retry again and often immediately within a short
time delay. Hence, the Greylisting concept includes the idea of using
a "Blocking Time" factor where a greylisted recorded mail sender is
blocked for a certain time period. Only when the blocking time has
expired, will the GreyListing server finally allow the mail sender to
continue with the transaction.What sort of impact has Greylisting had with Mail Delivery?Greylisting has been designed since its conception to satisfy
certain criteria:Enforce SMTP compliance with expected SMTP retry
strategies,Limit abusive mail senders ability to circumvent the
blocking,Have minimal impact on users, andRequire minimal maintenance at both the user and administrator
level.The first immediate impact are the increasing delays in mail
delivery due to the wide range in Greylisting blocking time values
which can be seconds, minutes to hours. Since SMTP has a standard
recommendation to implement a Progressive Retry queuing strategy (see
section 4.5.4.1 in RFC5321) where the
first few attempts have short delays (i.e. two attempts within the
first hour) with a progressive back off longer delay before the
maximum attempts (i.e. over 4-5 days) are exhausted, there are
increasing wasted attempts and foremost higher delays in delivering
mail. When a SMTP client implements an initial retry lower than the
remote GreyListing Server blocking time, the SMTP client will have
increasing wasted attempts overhead. When the SMTP client implements
an initial retry delay higher than the remote GreyListing Server
blocking time, the SMTP client will have unnecessary wasted mail
delays in delivering mail.With the increasing deployment of Greylisting mail servers, the
second impact is such that even the SMTP server who does not employ
Greylisting, will more than likely increasingly connect to a remote
mail server that does employ Greylisting and will experience the
temporary rejection overhead requiring additional mail sending
retries.The third impact is that many GreyListing servers now use the
rejection idea at the connection level using a 421 greeting response
which may be a different retry condition than a 45z rejection response
issued at the MAIL FROM or DATA state. Since many MTA clients see a
421 as a possible loading limit, it may use this to immediately
reschedule a retry using a different MX/IP host..Overall, Greylisting was designed to address the high abuse of
"single shot" anonymous mail senders, however it was done at the
expense of legitimate mail senders experiencing wasted mail attempts
and increasing delivery delays and with improper GreyListing server
and client settings, SMTP clients may now have to revisit their
queuing strategies to address the Greylisting overhead related
issues.This specification provides insights into preparing a low impact
Greylisting Server by providing some recommendations for blocking
delays and defining a formal structure GreyListing server to
optionally include a suggested "retry=time-delay" information in the
server's 4yz temporary text responses.This specification does not attempt to alter existing IETF standard
SMTP and non-IETF standard Greylisting protocols other than to provide
augmented Greylisting techniques to help alleviate the overhead
associated with Greylisting in the client/server SMTP transport
process. SMTP servers supporting this extension will only be altering
4yz greylisting responses which is out of scope in RFC5321.
Greylisting is not part of SMTP and is implemented as an "add-on"
component. SMTP clients supporting this extension will only be
factoring in a new time factor for their existing retry and queuing
method where the exact retry and queuing methods in placed is also out
of scope in RFC5321.The Greylisting method specified in this document is a
complementary method to any other existing mail filtering control
systems, and is not intended as a replacement for those other methods.
In fact, it is expected that abusive mail senders will eventually try
to minimize the effectiveness of this method of blocking, and
Greylisting is designed to limit options available to the mail senders
when attempting to do so. The positive outcome of Greylisting is that
the only methods of circumventing it will tend to make other mail
filtering control techniques just that much more effective (primarily
DNS and other methods of blacklisting based on IP address) even after
any adaptation by the abusive mail senders has occurred.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.Mail Transfer Agent. Sender or Receiver of mail.
Generally viewed as a router within a MSA intra-network where
there is a inherent authentication.Mail User Agent. Online or offline mail
reader/writer software. Typically has its own MTA component for
sending mail.Mail Submission Agent. Generally associated with
a MUA sending message to a ISP or ESP where there is an authorized
or authenticated association with the MUA.Mail Destination Agent. Generally associated as
the final destination of the message where the message is
typically targeted for a local user. If the MDA is going to route
the mail, then its behaving more as a MSA or MTA.This specification uses the Augmented Backus-Naur Form (ABNF) notation for the formal definition of
the syntax for the "retry=time-delay" hint.The following are out of scope considerations in this
specification:o how Greylisting information is recorded in databases,o what additional mail information is recorded in databases
beyond the Triplet recording, ando server reasons for an 4yz response outside a Greylisting
reason, such as SMTP Traffic Control concepts.The basic idea of GreyListing is:MTA Client initiates a mail delivery attempt to a remote
GreyListing compliant mail receiver (MDA),The GreyListing Server collects first time session information
about the sender such as connection IP, MAIL FROM and RCPT TO called
the Triplet.If the Triplet was never recorded before, the Triplet is recorded
and a 4yz rejection server response with a recommended
"retry=time-delay" hint is issued where the time reflects the
blocking time the sender can attempt again and proceed with the
transaction.If the Triplet was recorded, a check is performed to determine if
the blocking time has expired. If not, another 4yz rejection
response with a new "retry=time-delay" hint reflecting the new
blocking time is issued.When the sender tries again with the same recorded information
after the blocking time has expired, then the sender has passed the
server's greylist test and is allowed to proceed to send the
mail.One of the essential goals of this specification is to reduce the
network and communications overhead in sender attempts and to reduce
mail delivery delays by implementing the server "retry=time-delay" hints
in the 4yz greylist responses.In the classic Greylisting protocol described in HARRIS, a Triplet is the unique combination of
connection IP, the reverse address (MAIL FROM) and the forwarding
address (RCPT TO) used to track the sender. When the sender retries
with the same triplet, a lookup can be perform to determine its
Greylist status. However, depending on the Greylist server
implementation, it can reject at different points in the SMTP state
machine and may not collect the entire triplet information.While it is out of scope how a SMTP session Triplet is collected
and what SMTP session data points it contains, the key point is a
specific Triplet used to track the MTA for an initial transaction
attempt and subsequent retries in order to control it during the
Greylisting Server blocking time.The following is an implementation example for triplet
recording:One example of a triplet-alg is using a standard hashing algorithm
as such SHA1 with BASE32 encoding.BASE64(SHA1(CIP, RPATH, FPATH))Other tracking methods such as index keys in SQL database
tables are often common with Greylisting server implementations. This
specification does not define an formal triple-alg method. Any SMTP
data can be used as long as it represents Greylisting servers method
for consistent tracking transactions , its initial rejection and
subsequent acceptance with expected retries.Greylisting assumes a triplet recording (IP, FROM and TO), however
a Greylisting server can reject at any point in the SMTP state machine
by recording less information about the sender. This specification
hopes to assist the MTA to determine when a temporary rejection is
greylist related apart from other reasons which can be a factor in how
an MTA client will reschedule new attempts.Many SMTP servers will use a 421 response during the greeting as
a way to limit connections and control load.A GreyListing server deciding to greylist a client at the
connection greeting MUST use a 421 reply code and SHOULD include a
"retry=time-delay" hint as part of the text response.The "retry=time-delay" hint will help the MTA decide what sort of
rejection was imposed by distinguishing between loading limit or
greylist rejection. Without the "retry=time-delay" hint, a MTA can
try an alternative MX immediately (without delay) and the rejection
may still occur. Including the "retry=time-delay" hint will assist
the MTA to better reschedule the retry.A GreyListing Server rejecting at the connection level is
recording only the connection IP to track the sender.A GreyListing server deciding to greylist a client as a response
to the EHLO or HELO command SHOULD use a 451 reply code and SHOULD
include a "retry=time-delay" hint as part of the text response. The
hint will help the MTA decide when a new attempt should be
attempted.A GreyListing Server rejecting at the EHLO is recording the
connection IP and EHLO/HELO machine host name.Note: The editor has no information on the existence of
Greylisting servers that perform a 4yz rejection at the EHLO or HELO
command for greylisting reasons.A GreyListing server deciding to greylist a client as a response
to the MAIL FROM command SHOULD use a 451 reply code and SHOULD
include a "retry=time-delay" hint as part of the text response. The
hint will help the MTA decide when a new attempt should be
attempted.A GreyListing Server rejecting at the MAIL FROM is recording the
connection IP and MAIL FROM sender address.A GreyListing server deciding to greylist a client as a response
to the RCPT TO command SHOULD use a 451 reply code and SHOULD
include a "retry=time-delay" hint as part of the text response. The
hint will help the MTA decide when a new attempt should be
attempted.A GreyListing Server rejecting at the RCPT TO is recording the
connection IP, MAIL FROM and RCPT TO addresses.A GreyListing server deciding to greylist a client as a response
to the DATA End of Data (EOD) SHOULD use a 451 reply code and SHOULD
include a "retry=time-delay" hint as part of the text response. The
hint will help the MTA decide when a new attempt should be
attempted.Generally, a GreyListing server will allow the DATA command in
order to capture the actual RFC5322
message before a greylist response is issued. The reasons are beyond
the scope of this specification.A GreyListing Server rejecting at the DATA may be recording more
information besides the triplet information, i.e. Message-Id
header.Many current Greylisting Servers use varying text responses with
informal language try again time text information. The following are
known forms of existing Greylisting Servers which expose a form of
time hints within the text response:421 This server implements greylisting, please try again in #
seconds450 4.7.1 <RCPT>: Recipient address rejected: Greylisted
for # minutes450 4.7.1 <RCPT>: Recipient address rejected: Greylisted
for # seconds451 4.7.1 Greylisting in action, please come back in
HH:MM:SS451 Greylisted for # seconds451 Greylisted, please try again in # seconds451 Greylisting enabled, try again in # minutesIt is possible for existing MTA clients currently supporting the
parsing and extraction of the time factor with the known informal
responses from existing Greylisting servers and this specification
does not attempt to limit specific MTA client implementations which
may already exist.This specification offers a formal structure the Greylisting Server
MAY use within their 4yz responses and the MTA client MAY use to
detect and extract the retry information consistently without error
using a single format within the 4yz response containing the following
structured "retry=time-delay" tag:retry=[DD-]HH:MM:SSThe [DD-]HH:MM:SS part is the time delay the MTA SHOULD wait before
attempting to send the mail again. It is not a specific time of day,
but rather the amount of GreyListing Server blocking time expected by
the server before the client SHOULD try again. An MTA client ignoring
this information, attempting again before the blocking time has
expired, is a wasted attempt and can delay the mail delivery well
beyond the GreyListing server blocking time.In ABNF, GreyListing server response
syntax is:Examples:For multiple lines responses, the retryhint MUST be provided in the
last line of the response.GreyListing Servers may issue 421 or 45z responses at any point
in the SMTP session. However, RFC5321 recommends 421 be used at the
greeting and for server interruption events. This specification
recommends keeping with the SMTP RFC5321 recommendations for 421 and
only use 45z for non-Greeting rejections responses. All SMTP
compliant MTA will always follow 4yz for scheduling a retry, but the
difference is a 421 can trigger an immediate retry attempt without
delay at the next MX IP address, if any, where a GreyListing server
will most likely reject the new attempt due to the blocking
time.IMPLEMENTATION NOTE: RFC5321 recommends a specific 450 reply
code for temporary rejections related to local policy reasons.
HARRIS used 451 to make it distinctive as a greylist response.
This specification recommends using 450, however, it is
recognized that many existing Greylisting servers already use
451 as the reply code. MTA MUST NOT depend on 450 or 451 to make
retry decisions. All 4yz responses MUST be interpreted as a
temporary rejection.When the "retry=time-delay" hint is implemented in the
response, compliant MTA will be able to determine the difference
between a load restriction and a greylisted rejection to
appropriately reschedule a new attempt at the GreyListing server's
suggested time hint.This specification does not impose any specific blocking delay
value when 4yz rejections are issued by servers, other than to suggest
that timely delivery of mail to users remains to be an inherent
expectation by SMTP clients and SMTP servers.The GreyListing server blocking times vary greatly in practice, but
there is empirical evidence a majority of systems use a 1 to 5 minute
delay. Many use 10 minutes or 15 minutes. Many use less than 1 minute,
like 30 to 55 seconds. The latter tend to be systems who wish to lower
impact with immediate and timely mail delivery delays. However, this
can be wasteful attempts when the MTA is operating blindly with
unknown blocking times imposed by Greylisting Servers.When it comes to a recommendation, there is no GreyListing logic to
suggest that long delays be use when the goal of Greylisting senders
is to address the anonymous random "single shot" senders where their
triplet will never be the same. Delaying good SMTP senders for
extended unreasonable periods defeats the goal of Greylisting.Since there is no clear recommendation for a blocking time delay
(other than to keep it short as possible), this specification offers
the "retry=time-delay" hint as a method to alleviate the uncertainty
in the wasted attempts and delays in timely mail delivery.GREYLIST is a new ESMTP service
keyword. The GreyListing Server MAY add this optional keyword as a
response to EHLO command. EHLO response Format:250-GREYLIST [server-options]If the GREYLIST keyword is presented as part of the EHLO response, it
means the server has Greylisting implemented and 4yz responses are
possible due to a Greylist decision by the server to impose on the
client. The keyword is not necessary and the server can still provide
4yz temporary rejections.The optional server-options provides space separated attributes
reflecting the server Greylisting information the server wishes to
expose. Currently the following optional attributes are defined:means that 4yz responses related to GreyListing
will have "retry=time-delay" information. The attribute is optional
and not required to issue 4yz responses with "retry=time-delay"
hints.The SMTP server MAY add support for the GREYLIST service keyword
in the EHLO response. If the SMTP server adds the GREYLIST service
keyword without the RETRY attribute, it MAY add the
"retry=time-delay" hint to 4yz responses. If the SMTP server adds
the GREYLIST service keyword with the RETRY attribute, it MUST add
the "retry=time-delay" hint to to 4yz responses.The SMTP client MAY read the GREYLIST service keyword exposed by
the EHLO response and it MAY support the usage of the
"retry=time-delay" hint in 4yz responses and are not obligated to
honor the SMTP servers recommended retry delay.If the SMTP server offers the GREYLIST keyword with the RETRY
attribute, the SMTP client SHOULD consider supporting the usage of
the server's recommended retry delay in 4yz responses with
"retry=time-delay" hints.If a SMTP client is rejected by the Greylisting Server during the
session, the client SHOULD NOT attempt to start a new transaction
during the same session and SHOULD immediately issue a QUIT command
to end the session. Its been observed that some mail senders will
hold the connection for 1-5 minutes and retry the same mail
transaction or a new transaction. The SMTP server rejecting the
initial transaction MAY stop accepting any new transactions attempts
during the same session.If a SMTP server offers a "retry=time-delay" hint which results
in a wasted 2nd attempt and requires additional attempts, the SMTP
client MAY begin to ignore the server's "retry=time-delay" hint
after the 2nd wasted retry. The SMTP client implementation can
decide what limits to place on honoring "retry=time-delay" hints and
wasted attempts it provides.Example with no extended codes:Example with extended codes:Example of connection rejection:This document makes no request of IANA.Note to RFC Editor: this section may be removed on publication as an
RFC.One possible security concern envisioned is a DoS attack when
"retry=time-delay" information is exposed by the GreyListing server
where by a malicious sender may attempt to overwhelm the server during
the server's retry time exposing a time window when the server has
indicated system availability for mail acceptability. However, since
security measures to mitigate DoS is a required operational factor, a
GreyListing Server will inherently be prepared for DoS attacks with
managed loading limits with or without "retry=time-delay" Greylist
responses, thus there is no expected technical concern by exposing
Greylist "retry=time-delay" hints. With or without this specification,
all SMTP servers SHOULD be prepared for DoS attacks of all kinds.Another arguable security concern is related to the idea a formal
SMTP extension can possibly lower the effectiveness of Greylisting when
abusive mail senders adapt to the server's suggested retry times. This
concern does not seem to have weight since adaptation can occur with or
without the extension simply by complying to SMTP retry recommendations.
Greylisting remains effective because legacy abusive systems do not
adapt. In fact, a "retry=time-delay" hint implementation provides a
means to help avoid abusive redundancy and reduced random overloading of
connections at unmanaged random times by MTA clients of all flavors. A
"retry=time-delay" hint may actually be purposely calculated to provide
a time window when there is less loading for legitimate and abusive
senders.The following individuals contributed input and guidance in the
production of this specification:Claus Assmann, Frank Ellerrman, Tim Kehres, John Klensin, S.
Moonesamy, Keith Moore, Ken Raeburn, Paul Smith.Please note acknowledgement does not imply any specific
endorsement of this specification other than they have provided
important pros and cons input which helped mold the specification.The Next Step in the Spam Control War: GreylistingGreylisting Server implementations vary in ways which may include
many factors including how senders are traced, accepted, how record
expires, the history of sender transactions and including but not
limited to how senders are temporarily or permanently white listed. The
original Harris Greylisting specifications
offers a range of ideas that are considered. The following are just of a
few of additional parameters that are considered by servers:Whitelist Record Expiration:Whitelist
Record Expiration is used to allow a previous greylisted sender a
time window where it can be temporarily or permanently whitelisted
depending on the implementation. This is a local policy
consideration, however, it should be noted that redundant
greylisting of a common MTA is not considered reasonable. At some
point, the MTA is a trusted source of mail and the MTA SHOULD be
permanently whitelisted. The main idea with a temporary whitelisting
is that its possible a future transaction can be a compromised user
transaction.Class C IP Address Tracking:Class C IP
Address Tracking allows a Greylisting server to control a greylisted
MTA who retries using a different class C address. This is typical
in larger outbound farms where many machines are used to send mail.
If Class C is not considered, MTAs using a different IP will be
unnecessarily rejected after delaying within a blocked time.It is possible for a GreyListing server to combine other mail
filtering techniques, methods and session information to determine if a
sender should be greylisted. While the augmentation of these additional
methods is out of the scope, the following are some suggestions that may
help minimize a GreyListing Server impact. on MTAs.SPF (Sender Policy
Framework) can be used to help validate a sender's IP
association with the return path domain. A SPF SOFTFAIL or FAIL (if
not used for rejection) result could be used to help decide when
Greylisting should be employed on the sender. While a PASS result is
not a trusted condition, a local policy may use a PASS to skip
Greylisting mail checks.DKIM (Domain Key
Identified Mail) can be used to help authenticate the
transactions from trusted DKIM mail signers. If the signer is
considered is trusted source, this can help eliminate the need to
greylist the sender.Possible section showing real proof of concept examples.Review the SMTP Service Keyword and determine how SHOULD|MAY|MUST
is applied.