Subject: revised draft-malamud-no-soliciting-05
From: Carl Malamud <[email-masked]>
To: [email-masked]
Date: Sat, 31 Jan 2004 10:37:50 -0800 (PST)
List-Id: <ietf-smtp.imc.org>

Hi -

Based on comments received, I've put a new draft at:

http://trusted.resource.org/no-solicit/

It has also been resubmitted to the i-d queue.  Thanks
for everybody for your comments.  Please feel free to
send additional comments to this list, to me, to the
iesg, or to the forum of your choice.

Regards,

Carl

Date: Sat, 31 Jan 2004 12:25:58 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: Carl Malamud <[email-masked]>, [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

[email-masked] wrote:
> On Tue, 27 Jan 2004 06:15:11 PST, Carl Malamud said:
> 
> 
>>1. The fact that esmtp had no way for the receiver to indicate
>>that they don't want to be solicited.
> 
> 
>>2. The fact that our mail headers are being littered with
>>insertions by filtering software and, increasingly with obscure
>>markings like "ADV" required by policy makers.
> 
> 
> I'm still puzzled by what problem we actually expect this to solve.

As am I.  I strongly suspect that it won't solve the spam problem.  I'm
pretty sure that restricting the scope to ESMTP (i.e. ignoring plain SMTP)
is a mistake.  I share the doubts raised by others about whether an ESMTP
server or its operator can make blanket assertions about all of its
recipients.  RFC 2821 has made EXPN and VRFY less than mandatory due to
abuse in harvesting email addresses; providing a back door for address
harvesting ("Alice, Bob, and Carol...") is a questionable move in light
of the changes to EXPN and VRFY.  I fail to see how adding yet another
header field (i.e. in addition to Keywords) to convey originator-supplied
keywords is going to have any effect on reducing "litter" added by filtering
software -- indeed such a redundant header field could be considered
more litter.  And ignorant policy makers will continue to attempt to
impose silly requirements which will be ignored in practice no matter
what we do.

No SMTP extension can by itself reduce spam. A spammer can always circumvent
such a mechanism by starting a session with HELO rather than EHLO, in
which case extensions simply are not involved. Case closed.  At best, an
extension might serve to in some way tag legitimate mail. Given the fact
that for many email users the quantity of spam already significantly
exceeds that of legitimate mail, addressing the issue of tagging
legitimate mail seems like the path of least resistance and best
efficacy -- certainly relying on accurate sender-provided labels on
spam is a lost cause (it's not "solicitation" it's INFORMATION).

If there is a clear statement of a problem, I can't find it in the
draft.  Searching the draft for "problem" yields:
1. some text about legislation in Australia indicating that a certain
   activity is NOT a social "problem" there.
2. a statement that "This proposal is not meant to solve the UBE problem"
3. "solving the same problem
   addressed by this service extension."  We still have no idea what that
   "problem" might be.

And that's all.  If the author wishes to pursue this draft further, it
should be rewritten to provide a clear statement of the problem that it
is intended to solve, near the beginning of the document.  It should be
clear from reading the document that any requirements imposed by the
text contribute to solving that problem.  If it's not self-evident, there
should be some discussion illuminating the contribution to the solution
of the specified problem.  All of the (frankly, tedious in a technical
document) anecdotes should be removed or at least relegated to an Appendix.

More comments on recent discussions:

Carl Malamud:

>> > Yes.  Relays MUST simply relay messages.  Only when a message
>> > has reached an MTA under the *direct control of the recipient*
>> > can other actions be taken.  Is everybody happy with that
>> > phrase before I start sprinkling it throughout the draft?  :)
>> 
>> I don't mind the phrase, but you need to define it somewhere.  For 
>> instance, does "direct control of the recipient" allow an ISP to make 
>> assertions on behalf of its customers?  Does it allow a business's SMTP 
>> server to make assertions on behalf of its employees?
>> 
> 
> Oof.  No way am I going to try and define that concept.  :)
> 
> How about a reference to section 7.7 of 2821, "Scope of
> Operation of SMTP Servers"?  That seems to cover this 
> situation pretty carefully and I can quote the whole
> section (it only has 2 paragraphs).

First you need to decide whether or not it's relevant to say anything at
all about relaying. If the proposal is about relaying (at least in part),
then you need to decide what it is you want to say about relaying. RFC 2821
section 7.7 certainly doesn't say "Relays MUST simply relay messages". In
fact it gives relay operators quite wide scope in deciding what to relay
and what not to relay.

> This proposal is about the
> conveying of keywords *not* the decision to accept or
> reject messages.  That is, potentially, the subject
> of other standards or laws, but this proposal is a
> mechanism for the message sender and others involved
> in the delivery of the message to assert keywords,
> and for the recipient to assert keywords.  These
> words are all drawn from a registered list.

Then why not simply register keywords (for use with the existing Keywords
header field) and leave it at that?  Transport (SMTP/ESMTP) has no
business in poking about in the message content (and I don't want to give
Big Dumb Fascist Brother or his minions any excuse for doing so).

> to say
> "recipients Alice, Bob, and Carol also refuse to accept ADV messages"

That's the back door for purveyors of "20 million verified email addresses".

But there's a more fundamental problem. The draft states:

>  In the case of the
>    "PER-RECIPIENT" option, this information is not present in the
>    initial exchange and is instead presented as part of the "MAIL-FROM"
>    command.

RFC 2821 gives SMTP command sequence as first MAIL FROM, then RCPT TO.
How is an SMTP server supposed to give a per-recipient response at the
MAIL FROM stage, when recipients have not yet been specified?!?  It
turns out that the draft later contradicts itself by giving an example
that shows the response after RCPT TO,not MAIL FROM (which contains
no hyphen, by the way).

Arnt Gulbrandsen:

> If I'm wrong, NO-SOLICITING at least shouldn't hurt.

Any SMTP extension that doesn't solve a specific problem (or at least
contribute significantly to doing so) does hurt.  It causes people to spend
time on fruitless endeavors that could more productively be spent on
effective solutions to real problems.  It causes bloat in SMTP server
software.  If it also provides a back-door entry to private information,
it is definitely harmful.  If it turns a 1k spam message into a 1.1k
spam message it makes the spam problem bigger.  If it appears to give a
naive user reason to complain to/about a joe-job victim, it's harmful.
If it contradicts established SMTP protocol requirements or causes
confusion about those requirements, it is harmful.

Frankly, I see nothing useful in the proposal as it is currently
presented, and a great deal of potential harm in a number of areas.

> Parsing Received: and adding the information to SMTP-level arguments is bad, because it adds complexity: If Received: can be parsed, the recipient MTA/Sieve/MUA can do it and there's no need for RCPT TO arguments in the first place.

That's a big "if"; there are already a number of widely-deployed MTAs that
generate unparseable Received fields.  It's unclear if that can be fixed.
Received was, according to RFC 1123, primarily intended for human use in
manually tracing messages.  As it stands, there is little information in
Received fields which is simultaneously machine-readable, useful, and reliable.
The machine-readable trace information includes the easily-forged (and therefore
unreliable and useless) HELO/EHLO name, whereas the reliable information
(client IP address) is relegated to a comment, if present at all. Perhaps
that can be fixed. Perhaps Received needs to be replaced. I'm not sure. But
I am certain that adding additional cruft to Received isn't going to improve
matters; it will simply provide greater scope for incompetent implementors to
generate unparseable fields.

RFC 1123 perhaps erred in failing to specify a machine-readable mechanism
for recording the client IP address in a Received header field.  RFC 2821
attempts to impose structure on a comment in order to provide for recording
client IP address.  However, a comment is just a comment, and is not a
machine-readable part of a structured header field; it has the semantics of
whitespace (RFC 822).  This draft attempts to impose semantics on comments
also, and that is doomed to failure.

Moreover, the example given in the draft lacks the mandatory "from" component
which an SMTP MTA is obliged to include.

Summary: the author should reflect on exactly what problem he is attempting
to solve, and formulate a clear and concise statement of that problem.  He
should then consider whether an SMTP extension is capable of solving that
problem, including the applicability of extensions to the problem domain
in the presence of widespread use of unextended SMTP.  If he still feels
that an SMTP extension can solve the problem, he should provide a clear and
concise specification for such a mechanism and he should provide an
explanation of how the mechanism proposed in fact solves the specified
problem.  He should pay particular attention to the specification of the
SMTP protocol and the text message format (RFCs 822 and 2822), ensuring
that his proposal does not conflict with those standards, and he should
examine possible security and privacy issues related to his proposal.

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard 
From: [email-masked]
Date: Sat, 31 Jan 2004 12:36:29 -0500
Sender: [email-masked]

--==_Exmh_1776197491P
Content-Type: text/plain; charset=us-ascii

On Sat, 31 Jan 2004 03:10:45 EST, Hector Santos said:

> They may not support this specific proposal from the "complex" standpoint of
> the extra requirements,  SOLICIT=,  Solicitation: header,  etc, but they
> might use the basic "NO SOLICITATON" seen on the "door" as they pass by the
> store, to not enter the store.  After all, isn't this one of the basic
> premise and human analogy used for this proposal?

The difference is that in the liveware world, it's fairly easy to phone the
local constables and have them have a chat with the people involved before
they've visited more than 3 or 4 additional establishments.

--==_Exmh_1776197491P
Content-Type: application/pgp-signature

--==_Exmh_1776197491P--

From: "Hector Santos" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>, <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>, "Carl Malamud" <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Sat, 31 Jan 2004 03:10:45 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I agree.  It won't hurt to have it.

My only thought is the consideration on how to help the idea of "auto client
detection" by advertising it more upfront (i.e, greeting).

They may not support this specific proposal from the "complex" standpoint of
the extra requirements,  SOLICIT=,  Solicitation: header,  etc, but they
might use the basic "NO SOLICITATON" seen on the "door" as they pass by the
store, to not enter the store.  After all, isn't this one of the basic
premise and human analogy used for this proposal?

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Arnt Gulbrandsen" <[email-masked]>
To: <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>; "Carl Malamud"
<[email-masked]>
Sent: Friday, January 30, 2004 2:54 PM
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed
Standard

>
> [email-masked] writes:
> > Is there a significant number of spammers that would actually take the
> > time to look for a No Soliciting marker, and actually abide by it?
>
> Mainsleaze only, I think.
>
> So, is there a significant amount of mainsleaze? I think yes. Not major,
> but signficicant.
>
> That's just the view from where I'm sitting. If I'm wrong, NO-SOLICITING
> at least shouldn't hurt.
>
> Arnt
>
>

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Cc: Keith Moore <[email-masked]>, [email-masked], Carl Malamud <[email-masked]>
Date: Fri, 30 Jan 2004 20:54:03 +0100
Sender: [email-masked]

[email-masked] writes:
> Is there a significant number of spammers that would actually take the 
> time to look for a No Soliciting marker, and actually abide by it?

Mainsleaze only, I think.

So, is there a significant amount of mainsleaze? I think yes. Not major, 
but signficicant.

That's just the view from where I'm sitting. If I'm wrong, NO-SOLICITING 
at least shouldn't hurt.

Arnt

Date: Fri, 30 Jan 2004 12:41:12 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Comments on and FWD: I-D ACTION:draft-klensin-emailaddr-i18n-02.txt
Sender: [email-masked]

--==========21B830F4BD724B916745==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi.  This draft is an update on the SMTP approach to mailbox 
internationalization.

Three things:

	* I am copying the SMTP list on this because everything
	is getting connected to everything else, as my on-list
	discussion with Nathaniel about the "trace fields in the
	envelope" proposal illustrates.  Please, to preserve
	everyone's sanity, do not start parallel discussions or
	cross-post -- this one is clearly an IMAA issue, at
	least for historical reasons.
	
	* There is one more proposal I-D coming in this series.
	As a one-line summary, it outlines ("specifies" would be
	too strong at this stage, but that is the intent) the
	encapsulation model that this draft suggests for
	downgrading.
	
	* Very high-level summary of changes from the -01
	version: I have become convinced that, if we are going
	to have an internationalized structure for email --not
	merely a collection of kludges and workarounds-- we are
	going to need to make some rather basic changes.  They
	include SMTP extensions for i18n addresses and alternate
	addresses and UTF-8 headers as well as UTF-8 headers, in
	some form, themselves.  I've picked up on Paul's UTF-8
	header proposal because it seems sensible and nothing
	else is on the table, but the proposal announced below
	eliminates the need for alternate address headers and
	all of the mucking around in the message payload by MTAs
	they imply.   And I'm convinced that we should have
	_one_ header to cover this rather than a collection.
	I.e., the model is that one is internationalized or not,
	rather than one in which internationalization is done
	piecemeal and incrementally, with all of the profiling,
	multiplicative cases, and long-term cruft that would
	imply.

Please do not react to what you think is in this proposal 
without reading it or, especially based only on what you think 
it is about from reading the above.   That is a waste of your 
time and that of everyone on the list, as previous rounds of 
such discussions amply demonstrate.

best,
    john

---------- Forwarded Message ----------
Date: Friday, 30 January, 2004 11:49 -0500
From: [email-masked]
To: IETF-Announce
Subject: I-D ACTION:draft-klensin-emailaddr-i18n-02.txt

A New Internet-Draft is available from the on-line
Internet-Drafts directories.

	Title		: Internationalization of Email Addresses
	Author(s)	: J. Klensin
	Filename	: draft-klensin-emailaddr-i18n-02.txt
	Pages		: 30
	Date		: 2004-1-29
	
Internationalization of electronic mail addresses is, if
anything, more important than the already-completed effort for
domain names. In most of the contexts in which they are used,
domain names can be hidden within or as part of various types of
references. Email addresses, by contrast, are crucial: use of
names of people or organizations as, or as part of, the email
local part is, for obvious reasons, a well-established tradition
on the network. Preventing people from spelling their names
correctly is, in the long term, inexcusable.  At the same time,
email addresses pose a number of special problems -- they are
more difficult than simple domain names in some respects, but
actually easier in others. This document discusses the issues
with internationalization of email addresses, explains why some
obvious approaches are incompatible with the definitions and use
of Internet mail, and proposes a solution that is likely to
serve users and the network well for the long term.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-klensin-emailaddr-i18n
-02.txt

To remove yourself from the IETF Announcement list, send a
message to  ietf-announce-request with the word unsubscribe in
the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with
the username "anonymous" and a password of your e-mail address.
After logging in, type "cd internet-drafts" and then
	"get draft-klensin-emailaddr-i18n-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-klensin-emailaddr-i18n-02.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail
readers 	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---------- End Forwarded Message ----------

--==========21B830F4BD724B916745==========
Content-Type: message/rfc822;
 name="I-D ACTION:draft-klensin-emailaddr-i18n-02.txt"

Delivery-date: Fri, 30 Jan 2004 12:19:38 -0500
To: IETF-Announce: ;
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-klensin-emailaddr-i18n-02.txt
Date: Fri, 30 Jan 2004 11:49:38 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: Internationalization of Email Addresses
	Author(s)	: J. Klensin
	Filename	: draft-klensin-emailaddr-i18n-02.txt
	Pages		: 30
	Date		: 2004-1-29
	
Internationalization of electronic mail addresses is, if anything, more important than the already-completed effort for domain names. In most
of the contexts in which they are used, domain names can be hidden within or as part of various types of references. Email addresses, by contrast, are crucial: use of names of people or organizations as, or as part of, the email local part is, for obvious reasons, a well-established tradition on the network. Preventing people from spelling their names correctly is, in the long term, inexcusable.  At the same time, email addresses pose a number of special problems -- they are more difficult than simple domain names in some respects, but actually easier in others. This document discusses the issues with internationalization of email addresses, explains why some obvious approaches are incompatible with the definitions and use of Internet mail, and proposes a solution that is likely to serve users and the network well for the long term.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-klensin-emailaddr-i18n-02.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-klensin-emailaddr-i18n-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-klensin-emailaddr-i18n-02.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-klensin-emailaddr-i18n-02.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-klensin-emailaddr-i18n-02.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

--==========21B830F4BD724B916745==========--

To: Carl Malamud <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard 
From: [email-masked]
Date: Fri, 30 Jan 2004 12:28:33 -0500
Sender: [email-masked]

--==_Exmh_-490700280P
Content-Type: text/plain; charset=us-ascii

On Tue, 27 Jan 2004 06:15:11 PST, Carl Malamud said:

> 1. The fact that esmtp had no way for the receiver to indicate
> that they don't want to be solicited.

> 2. The fact that our mail headers are being littered with
> insertions by filtering software and, increasingly with obscure
> markings like "ADV" required by policy makers.

I'm still puzzled by what problem we actually expect this to solve.

Is there a significant number of spammers that would actually take the
time to look for a No Soliciting marker, and actually abide by it?

Reading Section 3, there's a lot of uses of "reputable mailers", when
that's not the problem....

(Don't take the "We could pass a law banning the attempted bypass of
said banners" - given the number of spam that are outright fraud and
the lack of progress in enforcing existing laws, I don't see that as
being a useful direction.  So unless there's something more substantial
than "this time it will be different, I promise", a new law won't help).

--==_Exmh_-490700280P
Content-Type: application/pgp-signature

--==_Exmh_-490700280P--

Date: Thu, 29 Jan 2004 08:35:39 -0500
From: Keith Moore <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

> > I believe this is from an earlier draft.  Go reread the law as it was
> > actually passed by Congress.  I don't see IETF mentioned in that law.
> > Basically as I read the law, IETF isn't in the loop, and there's no
> > expectation that any protocol extensions are needed.
> 
> hmmmm I thought I had a reference the most current bill when it was
> finalized.. let me check again,
> 
> ok, I reference the wrong section.  It is section 11.2.

ah, thanks.  it must be that every time I read the bill, my eyes had glazed
over or my brain had siezed before I got to that point.

(as bad as they are, IETF documents are a model of clarity compared to this.)

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Thu, 29 Jan 2004 08:36:28 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Thursday, January 29, 2004 8:09 AM
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed
Standard

> I suspect lawmakers would be more sympathetic to the need for a NO
> SOLICITATION sign to keep people from being disturbed, than for one to
keep
> computers from being disturbed.  The actual recipients of the message can
> avoid being disturbed by filtering mail, if the mail is required to be
> labelled.  The SMTP extension is just a way to do that more efficiently.

I see your point.

> > In CAN-SPAM,  Section 10. STUDY OF EFFECTS OF COMMERCIAL ELECTRONIC
MAIL.
> > Item #2 says:
>
> I believe this is from an earlier draft.  Go reread the law as it was
> actually passed by Congress.  I don't see IETF mentioned in that law.
> Basically as I read the law, IETF isn't in the loop, and there's no
> expectation that any protocol extensions are needed.

hmmmm I thought I had a reference the most current bill when it was
finalized.. let me check again,

ok, I reference the wrong section.  It is section 11.2.

SEC. 11. IMPROVING ENFORCEMENT BY PROVIDING REWARDS FOR INFORMATION ABOUT
VIOLATIONS; LABELING.

(2) a report, within 18 months after the date of enactment of this Act, that
sets forth a plan for requiring commercial electronic mail to be
identifiable from its subject line, by means of compliance with Internet
Engineering Task Force Standards, the use of the characters `ADV' in the
subject line, or other comparable identifier, or an explanation of any
concerns the Commission has that cause the Commission to recommend against
the plan.

Still there Keith. You won't find acroymn IETF :-)

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Thu, 29 Jan 2004 08:09:45 -0500
From: Keith Moore <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

> > p.s. At some point lawmakers might decide that a client's failure to
> > honor NO SOLICITATION is a kind of denial-of-service attack, but I sort
> > of doubt it will ever get to that point.  I consider it somewhat more
> > likely that they would require senders to include Solicitation: fields
> > in their messages, and that they would define specific keywords to use
> > in those cases.  (let's hope that different legislative bodies don't
> > produce conflicting definitions of the same keywords)
> 
> 
> The only reason for this is to satisfy the CAN-SPAM Act or any other
> national law that mandates topic identification.  

I'm not disagreeing with you, just making a statement about what form I
think such laws would take (at least in the US).

I suspect lawmakers would be more sympathetic to the need for a NO
SOLICITATION sign to keep people from being disturbed, than for one to keep
computers from being disturbed.  The actual recipients of the message can
avoid being disturbed by filtering mail, if the mail is required to be
labelled.  The SMTP extension is just a way to do that more efficiently.

> In CAN-SPAM,  Section 10. STUDY OF EFFECTS OF COMMERCIAL ELECTRONIC MAIL.
> Item #2 says:

I believe this is from an earlier draft.  Go reread the law as it was
actually passed by Congress.  I don't see IETF mentioned in that law.
Basically as I read the law, IETF isn't in the loop, and there's no
expectation that any protocol extensions are needed.

Keith

From: "Hector Santos" <[email-masked]>
To: "Carl Malamud" <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Thu, 29 Jan 2004 07:49:18 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Everyone,  The FTC wants input on this important topic of mail topic
identification:

http://www.ftc.gov/opa/2004/01/adult.htm    (copy of RFC below)

Carl,  this is right up your alley   Definitely something (new class) to add
to your specification.   Unfortuntely, IMO, as I  mentioned in mail,  it
seems the "Subject" line will be the most natural line where this
information will be important.  Spammers will view a Solicitation: line as
redundant.

If any, we have two weeks to provide IETF input.

My input.

It talks about this "Brown Paper Wrapper"   Is this some layman term for
SMTP envelope?

Since mail filtering methods is available today at the MUA end point to help
support this via rule-based mail filtering rules, the IETF needs define how
enforcment policies that will be made available at the transport level.

If this is going to be Carl's proposal,  IMO,  we need to decide on this
particular question:

Will client software be require to support the server advertisment policy
(EHLO No-Solicitation)?   In order words, if the client intention is to send
spam, it is require (MUST? SHOULD?) to read this EHLO tag and issue a QUIT
to end the session.    This falls into two categories.

Types of spammer clients:

1) Spammers might argue they are usng 3rd party mail delivery SMTP service
bureaus hence are complying with the SUBJECT line only and have no control
over 3rd party software compliance with the new EHLO and/or envelope.

2) Spammers with internal, proprietary in-house bulk send mail SMTP clients
with the ability to change the software to comply with the new EHO and/or
envelope.

Must they comply?

How about transition time frame for client software?

Anyway..... Here is a copy of the RFC (Request for Comment)

For Release: January 28, 2004

FTC Seeks Public Comment on Adult Content Spam Rule

As required by the CAN-SPAM Act, the Federal Trade Commission today
announced a proposal to establish a mark or notice that will be required for
inclusion in spam that contains sexually oriented material. The purpose of
the mark or notice is to inform the recipient that a spam message contains
sexually oriented material and to facilitate filtering of such spam
messages. Establishment of a mark to accomplish these purposes is one of
several actions that Congress has directed the Commission to undertake by
enacting the CAN-SPAM Act, which was signed into law on December 16, 2003.

A Federal Register Notice to be published shortly will seek public comment
on the proposal. The CAN-SPAM Act requires the Commission to prescribe the
mark or notice within 120 days after passage of the Act. Because of the
statutory deadline, the comment period ends on February 17.
The FTC proposes to adopt a rule prescribing the phrase
"SEXUALLY-EXPLICIT-CONTENT: " as the mark or notice mandated by the CAN-SPAM
Act. The proposed rule also would follow the intention of the CAN-SPAM Act
to protect consumers from unwitting exposure to pornographic images in spam,
by requiring this mark to be included both in the subject line of any e-mail
message that contains sexually oriented material, and in the electronic
equivalent of a "brown paper wrapper" in the body of the message. This
"brown paper wrapper" would be what a recipient would initially see when
opening a message containing sexually oriented material. It would include
the prescribed mark or notice, certain other specified information, and no
other information or images. The proposed rule would include a "definitions"
section to clarify that certain terms taken from the Act and appearing in
the rule have the definitions prescribed by particular referenced sections
of the Act. Finally, the proposed rule would include a severability
provision, so that in the event a portion of the rule is struck, the
remainder of the rule will stay in effect.

The Commission vote to approve publication of the Federal Register notice
was 5-0.

Questions or comments can be directed to [email-masked] or to Jonathan
Kraden at 202-326-2614.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Carl Malamud" <[email-masked]>, "Harald Tveit Alvestrand" <[email-masked]>, "Arnt Gulbrandsen" <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Thu, 29 Jan 2004 05:57:19 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed
Standard

> p.s. At some point lawmakers might decide that a client's failure to
> honor NO SOLICITATION is a kind of denial-of-service attack, but I sort
> of doubt it will ever get to that point.  I consider it somewhat more
> likely that they would require senders to include Solicitation: fields
> in their messages, and that they would define specific keywords to use
> in those cases.  (let's hope that different legislative bodies don't
> produce conflicting definitions of the same keywords)
>

The only reason for this is to satisfy the CAN-SPAM Act or any other
national law that mandates topic identification.  Otherwise, why would
Spammers support it?

In CAN-SPAM,  Section 10. STUDY OF EFFECTS OF COMMERCIAL ELECTRONIC MAIL.
Item #2 says:

             2) a report, within 18 months after the date of enactment of
this Act, that sets forth a plan
             for requiring commercial electronic mail to be identifiable
from its subject line, by means of
             compliance with Internet Engineering Task Force Standards, the
use of the characters `ADV'
             in the subject line, or other comparable identifier, or an
explanation of any concerns the
             Commission has that cause the Commission to recommend against
the plan.

Assuming this particular proposed is endorsed,  what I envision when the
IETF reports to the FCC its final input CAN-SPAM Act refinement,  the direct
marketer lobbies might  resist it for the following arguments:

1) Complexity and high cost of implementation.

There are two parts that are basically defined:  Support for Server Policy
Detection at the protocol level and topic identification at MAIL FROM and
the body header.    It promotes costly client software changes deemed
unnecessary or more than required for CAN-SPAM compliance, hence a barrier
to acceptance and deployment. The router situation is important here.   They
might argue that only topic identification is necessary.  See #3

2)  Inherently and traditionally mail software use a SUBJECT line for topic
identification.

They might argue their CAN-SPAM Ready software is already using the Subject
line for multiple delivery systems and compliance with other laws that are
based on a natural usage of a subject line.  The solicitation line will be
deemed as unnecessary. Only applies to SMTP.  [Side note: isn't this a
Network Control Line?.  They might argue MUA software are not designed to
deal with network control lines]

3)  Easier Server designs:

If they are smart, they might argue that it is far easier and less costly
design change to add a new state such as SUBJ: for clients and servers. This
is especially true for routing situations.   They might be resistance to
this because this closes the current loophole where topic identification is
only possible when the DATA is received - a highly desirable state point for
spammers to reach.   But this all depends on the strength of the IETF
enforcement for topic identification.

Unless I am completely off base,  thanks to you,  <g>   the only value I see
in this proposal is the server no-solicitation advertisement in EHLO,  As
you said, that's an highly desirable optimization feature.

Finally, of course, none of this will not work unless the IETF takes a
stance on enforcement (i.e, not optional).   What part of the proposal
should be enforceable?   The Server Policy Identification or the Topic
Identification?  If this particular proposal is endorsed,  a server
advertising No-Solicitation finding a solicitation: header at the DATA stage
is grounds for automatic rejections, auditing and tracing and reporting to
the spam-cops because they didn't follow the EHLO tags?

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "Carl Malamud" <[email-masked]>
Cc: "Harald Tveit Alvestrand" <[email-masked]>, "Arnt Gulbrandsen" <[email-masked]>, "Keith Moore" <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Thu, 29 Jan 2004 04:59:23 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Carl Malamud" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Wednesday, January 28, 2004 8:47 PM
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed
Standard

> > Carl,
> >
> > The basic problem with it is that it REQUIRES passthru systems to
support
> > this.
> >
>
> Not true.  Read it again and look at the traffic
> on the list.  That wasn't my intent and, if in my
> draftmanship I didn't get it right, I believe we
> can make that all clearer.
>
> Can you give me a chance to revise based on the discussion
> we've heard and see if it addresses your concerns?
>
> Regards,

Hi Carl,

I have put much thought and time into this among other things. I have read
your specs over and over, like I had back in rev 01 and rev 02 and yes,  I
have read it again now and re-read the thread.

But you are the cog of this proposal, so obviously you have probably better
seen it thru.  If you believe I am wrong, by all means, please show me why.

The following is my technical view on this.  Lets take a stand back into the
balcony and look at what we have with SMTP today.  The SMTP original spirit
and fundamental ground rules are:

1)  Accept all final destination mail, and

2)  Restrict relaying to authenticated or trusted sessions only (established
by whatever means, IP relay tables, AUTH, POP B4 SMTP, etc).

That's pretty much it I believe. I think it was Henry Ford who said
"Fundamental, simple, anyone can drive it!"  <g>

So to me, if the sender is non-anonymous (trusted session), I really don't
give a hoot . The  trust is already established.  They can send whatever
they want.   The server implementation may or may not display the
NO-SOLICITATION.   I believe your specs (section 4) say the absence of the
tag "SHALL NOT" be construed as consent.   That's works for me. Its an
implementation issue. But I will like to get back to this later.

However,  when we have no trust, that is where the trouble begin.  The
current industry problem is a result of the lack of ability to reliably and
optimally "gain or obtain that trust" at the SMTP level.  I call this
anonymous access.

So in this vain,  anonymous access controls need to be place on what can be
delivered to the local user (either with system wide or per recipient based
on the implementation).   This and any other proposal in this area best
applies to anonymous access final destination mail only.

But now lets gets back to non-anonymous routed messages by spammers.

The problem I see is this Technical Software Design issue that MUST be
considered in the implementation.

When a CLIENT is compliant with CAN-SPAM (which you need to agree will be
the only reason why they may want to support this proposal in the first
place),  the receiver now routing the message will need to be COMPLIANT
whether or not the router and/or the downlink receiver have a
no-solicitation policy.

Why?

Because the router NEEDS to be ready compliant just in case the downlink
receiver is compliant.  If the downlink is not compliant, then that is not a
problem.

Look at it this way.

We have a compliant client and the compliant final destination receiver
host.  No routing. The sender is going direct to the final destination host.
The server's system admin policy expects and only allows "CAN-SPAM ready"
clients following your proposal to gain access.  The CAN-SPAM ready client
expectation is both technical and legal as well.

Nothing in the middle can alter this technical and legal expectation of the
client and final destination server.

So the only way around I see is as follows:

You must document a technical specification that  final destination SMTP
server software with protocol level dynamic no-solicitation accept/reject
logic, MUST support the compliant CAM-SPAM client by performing the
accept/reject logic at both the MAIL FROM (or RCPT) state and also the DATA
state in order to catch the possible solicitation: header.

If I am missing something, please tell me.

Please remember,  in my view, CAN-SPAM is the only reason this need is
required.  Spammers are not going bother otherwise.   CAN-SPAM ready clients
are going to make sure that any direct trust with a user is not broken by
some non-compliant middle ware.  So a final destination server can not
solely rely on just using the MAIL FROM SOLICIT= to reject/accept spam.  It
will need to also have a redundant logic at the DATA stage just in case a
non-compliant router broke that trusted spammer-user relationship.

Also, you do realized this isn't going to work unless the IETF mandates that
CAN-SPAM Ready clients follow this specification?    Which to me is another
monkey wrench because in my view, when this is endorsed,  I believes
spammers will resist it and reject it because it complexity creates a
deployment problem and they already use the SUBJECT line for topic
identification. They will scratch their heads on why a Solicitation line is
needed when it may be quite possible that they are using multiple and new
delivery systems down the road.

Anyway, thats my view on the matter.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Hector Santos <[email-masked]>
Date: Wed, 28 Jan 2004 17:47:46 -0800 (PST)
CC: Harald Tveit Alvestrand <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, Keith Moore <[email-masked]>, [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

[ Charset ISO-8859-1 unsupported, converting... ]
> Carl,
> 
> The basic problem with it is that it REQUIRES passthru systems to support
> this.
> 

Not true.  Read it again and look at the traffic
on the list.  That wasn't my intent and, if in my
draftmanship I didn't get it right, I believe we
can make that all clearer.  

Can you give me a chance to revise based on the discussion
we've heard and see if it addresses your concerns?

Regards,

Carl

Cc: Keith Moore <[email-masked]>, "Carl Malamud" <[email-masked]>, "Harald Tveit Alvestrand" <[email-masked]>, "Arnt Gulbrandsen" <[email-masked]>, <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Wed, 28 Jan 2004 20:33:16 -0500
To: "Hector Santos" <[email-masked]>
Sender: [email-masked]

As sort of an experiment I'm trying to think about it this way:

The real indication about sender intent is in the Solicitation: header 
field.  This is arguably where it belongs, because it's a description 
of the message content (or more specifically the intent behind the 
content) that isn't very related to mail transport.

The MAIL FROM SOLICIT= parameter and the system-wide policy 
announcement in the EHLO response are basically optimizations.  If the 
SMTP client realizes that a message won't get through anyway because 
the server is going to filter it due to the server's announced policy, 
it's better for both the client and the server if they terminate the 
session early.  Similarly if the server realizes that the sender's 
SOLICIT= parameter is incompatible with the policies of all of the 
message's recipients, it can avoid accepting the message at all - and 
by telling the SMTP client that up front, the error reporting becomes 
more reliable.

Seen from this angle, there's no requirement to "pass" SOLICIT at all 
from server to client, because it's not an end-to-end service.  Each 
MTA can elect to use the EHLO response, and (if the extension is 
supported) the SOLICIT option if it wants to, assuming that there's a 
Solicitation: field in the message.   (It's a lot like the SIZE 
extension in the sense that each MTA derives SIZE by looking at the 
message rather than looking at the SIZE parameter it got when the 
message was received.)

The trick to making this work is to REQUIRE the SMTP server that 
advertises NO-SOLICITATION to implement its policy consistently (based 
on the Solicitation: field, and also based on the SOLICIT= field if 
present) regardless of whether the client implements the optimization.  
So the result of trying to send a message is essentially the same 
either way; it just saves time and bandwidth to implement the 
extension.

Keith

p.s. At some point lawmakers might decide that a client's failure to 
honor NO SOLICITATION is a kind of denial-of-service attack, but I sort 
of doubt it will ever get to that point.  I consider it somewhat more 
likely that they would require senders to include Solicitation: fields 
in their messages, and that they would define specific keywords to use 
in those cases.  (let's hope that different legislative bodies don't 
produce conflicting definitions of the same keywords)

> The basic problem with it is that it REQUIRES passthru systems to 
> support
> this.
>
> If the router does not support the relaying of this SOLICIT 
> information,
> then we now have given a spammer a loophole, or put another way, the
> responsibility is now on the router to make sure the client WHO MAY BE
> compliant with CAN-SPAM continues to be compliant during a route.   A
> passthru system can not alter this, and in some vain can have a legal 
> impact
> in making what is otherwise a compliant client,  non-compliant during 
> the
> route.   Someone will take the blame.
>
> The only way this proposal works is for final destinations.  But the 
> client
> will not know if has reached the final destination or not.

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 28 Jan 2004 14:09:15 -0800 (PST)
CC: Arnt Gulbrandsen <[email-masked]>, Keith Moore <[email-masked]>, [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

> 
> 
> --On 28. januar 2004 18:27 +0100 Arnt Gulbrandsen 
> <[email-masked]> wrote:
> 
> > Keith Moore writes:
> >>>  Btw, I also suggest sticking in some text noting that supporting
> >>>  SOLICIT as server does not require supporting it as SMTP client.
> >>
> >> what purpose would that server, other than discourage servers from
> >> relaying SOLICIT parameters?
> >
> > I'm just tired of hearing variations of "PIPELINING is hard to implement".
> 
> so what you mean is that a client connecting to a SOLICIT server doesn't 
> have to do anything special - in particular, doesn't have to parse the 
> SOLICIT EHLO response unless it wants to send SOLICIT-flagged mail?
> 
> makes sense to me.
> 

Hi Harald -

You are correct.  We are *not* requiring the use of this extension
at any time.  We are only saying "if you are using this extension, here is
how it should operate."

Regards,

Carl

Date: Wed, 28 Jan 2004 13:02:50 -0800
From: Harald Tveit Alvestrand <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

--On 28. januar 2004 18:27 +0100 Arnt Gulbrandsen 
<[email-masked]> wrote:

> Keith Moore writes:
>>>  Btw, I also suggest sticking in some text noting that supporting
>>>  SOLICIT as server does not require supporting it as SMTP client.
>>
>> what purpose would that server, other than discourage servers from
>> relaying SOLICIT parameters?
>
> I'm just tired of hearing variations of "PIPELINING is hard to implement".

so what you mean is that a client connecting to a SOLICIT server doesn't 
have to do anything special - in particular, doesn't have to parse the 
SOLICIT EHLO response unless it wants to send SOLICIT-flagged mail?

makes sense to me.

                   Harald

Date: Wed, 28 Jan 2004 17:01:20 -0500
From: Keith Moore <[email-masked]>
To: Carl Malamud <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

> > > Yes.  Relays MUST simply relay messages.  Only when a message
> > > has reached an MTA under the *direct control of the recipient*
> > > can other actions be taken.  Is everybody happy with that
> > > phrase before I start sprinkling it throughout the draft?  :)
> > 
> > I don't mind the phrase, but you need to define it somewhere.  For 
> > instance, does "direct control of the recipient" allow an ISP to
> > make assertions on behalf of its customers?  Does it allow a
> > business's SMTP server to make assertions on behalf of its
> > employees?
> > 
> 
> Oof.  No way am I going to try and define that concept.  :)
> 
> How about a reference to section 7.7 of 2821, "Scope of
> Operation of SMTP Servers"?  That seems to cover this 
> situation pretty carefully and I can quote the whole
> section (it only has 2 paragraphs).

I guess I think it is a stretch to say that a site-wide policy of
filtering some messages based on sender-selected labels is an
"operational or technical reason".  Yes, there are unfortunately
cases when sites have legitimate reasons for refusing, or even 
discarding, mail that meets certain criteria - but I seriously 
doubt that sender-selected labels are useful criteria for choosing
to employ such drastic measures.  And while I do recognize that
some sites have valid reasons for setting policies on behalf of
all of their users, I don't want this document to encourage ISPs 
to do this.

But to me, "direct control of the recipient" means that the
recipient (NOT the site) chooses the criteria by which messages
are rejected.

I don't think the 2nd paragraph of section 7.7 is apropos here,
or at least, it's talking about something different then what I
thought we were talking about.  

I don't think this is quite there yet, but here's a stab at text:

> - Intermediaries that aren't acting on behalf of either the sender or
> recipient SHOULD NOT filter based on SOLICIT= or Solicitation: or on
> Received: fields
> 
> - A per-server policy SHOULD NOT be specified unless it is reliably
> known that the policy is appropriate for all users of that SMTP server,
> including users whose mail is locally delivered and users whose incoming
> mail is routed through the local server for delivery on other servers.
> 
> - A per-recipient policy SHOULD be under direct control of the
> recipient, meaning that the recipient has consented to the policy
> and has the ability to change or remove the policy.
> 
> It is recognized that there may be valid operational reasons for
> refusing mail based on its content.  As a general rule that any server
> MAY refuse to accept mail for any reason if there is a technical or
> operational requirement to do so (such as resource exhaustion or hostile
> attack) [ref 7.7 of rfc 2821].  However, the SMTP extension described
> in this document is not intended as a general-purpose mechanism for
> ameliorating operational difficulties and such mechanisms are outside
> the scope of this document.

Date: Wed, 28 Jan 2004 15:59:10 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

> Keith Moore writes:
> >>  Btw, I also suggest sticking in some text noting that supporting 
> >>  SOLICIT as server does not require supporting it as SMTP client.
> >
> > what purpose would that server, other than discourage servers from relaying
> > SOLICIT parameters?
> 
> I'm just tired of hearing variations of "PIPELINING is hard to implement".

it's a different problem, I think.  PIPELINING isn't an end-to-end service, 
so implementing just the server half of PIPELINING is perfectly reasonable.
OTOH, the carraige of SOLICIT wants to be an end-to-end service.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Date: Wed, 28 Jan 2004 18:27:59 +0100
Sender: [email-masked]

Keith Moore writes:
>>  Btw, I also suggest sticking in some text noting that supporting 
>>  SOLICIT as server does not require supporting it as SMTP client.
>
> what purpose would that server, other than discourage servers from relaying
> SOLICIT parameters?

I'm just tired of hearing variations of "PIPELINING is hard to implement".

Arnt

Date: Wed, 28 Jan 2004 11:04:04 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

> Btw, I also suggest sticking in some text noting that supporting SOLICIT 
> as server does not require supporting it as SMTP client.

what purpose would that server, other than discourage servers from relaying
SOLICIT parameters?

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Keith Moore <[email-masked]>
Date: Wed, 28 Jan 2004 07:50:49 -0800 (PST)
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked], [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

> 
> > Yes.  Relays MUST simply relay messages.  Only when a message
> > has reached an MTA under the *direct control of the recipient*
> > can other actions be taken.  Is everybody happy with that
> > phrase before I start sprinkling it throughout the draft?  :)
> 
> I don't mind the phrase, but you need to define it somewhere.  For 
> instance, does "direct control of the recipient" allow an ISP to make 
> assertions on behalf of its customers?  Does it allow a business's SMTP 
> server to make assertions on behalf of its employees?
> 

Oof.  No way am I going to try and define that concept.  :)

How about a reference to section 7.7 of 2821, "Scope of
Operation of SMTP Servers"?  That seems to cover this 
situation pretty carefully and I can quote the whole
section (it only has 2 paragraphs).

Regards,

Carl

From: Arnt Gulbrandsen <[email-masked]>
To: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Date: Wed, 28 Jan 2004 16:44:56 +0100
Sender: [email-masked]

Carl Malamud writes, answering me:
>>  I'd say that ONLY the first MTA may add SOLICIT=, and ONLY based on 
>>  the Solicitation field and/or the MUA's SMTP/SUBMIT SOLICIT= 
>>  argument. Other MTAs may not: if there are Received fields at all, 
>>  neither they nor Solicitation may be parsed to construct SOLICIT=.
>
> Hmmm ... I'd simplify that one more time. You need to be able to grab 
> the solicitation header in case there are non-supporting mtas in the 
> middle.

I considered that and IMO, that's not necessary. MTAs that are neither 
under control of the sender nor teh recipient generally only occur for 
mailing lists, and noone who's honest enough to use SOLICIT should be 
spamming a mailing list.

Btw, I also suggest sticking in some text noting that supporting SOLICIT 
as server does not require supporting it as SMTP client. For ten years, 
people have been assuming that supporting PIPELINING is hard, even 
though it's dead easy for the server.

Arnt

Cc: Keith Moore <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Wed, 28 Jan 2004 10:39:44 -0500
To: Carl Malamud <[email-masked]>
Sender: [email-masked]

> If you receive SOLICIT= using this extension, you must pass it
> along intact if the next hop supports this extension.
>
> If you receive a message from an mta that is not supporting this
> extension, you MAY (not must), for operational reasons,
> grab the keywords (all of them) from the Solicitation: header.

I have some doubts about this.  I understand the desire to speed 
deployment.  The problem is that to some extent the values that should 
go in the SOLICIT= and the Solicitation field should be dependent on 
context, and in particular, the set of recipients to whom a message is 
being sent.  The values in the Solicitation field might be correct when 
the message is originally sent to the set of recipients specified by a 
sender, but not valid when the message is resent to another party with 
the header intact.

I guess it boils down to a question of whether it's acceptable to copy 
SOLICIT= from Solicitation: field even on message submission.  If it 
is, then it will automatically be copied during a resend operation into 
that operation's envelope, and there's little harm to be done by having 
a subsequent MTA re-copy that field.

Overall I hate to encourage MTAs to do even more peeking into content 
than they do now - I think it's a trend in the wrong direction even if 
we do sometimes find it useful.

> Yes.  Relays MUST simply relay messages.  Only when a message
> has reached an MTA under the *direct control of the recipient*
> can other actions be taken.  Is everybody happy with that
> phrase before I start sprinkling it throughout the draft?  :)

I don't mind the phrase, but you need to define it somewhere.  For 
instance, does "direct control of the recipient" allow an ISP to make 
assertions on behalf of its customers?  Does it allow a business's SMTP 
server to make assertions on behalf of its employees?

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 28 Jan 2004 07:28:17 -0800 (PST)
CC: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

> > 1. The SOLICIT= parameter in the envelope MUST contain all 
> > solicitation class keywords found in the Solicitation header.
> >
> > 2. An optional TRACE= parameter MAY contain additional solicitation 
> > class keywords found in received: headers.
> >
> > I'm fine with that, but would like to hear from some others how they feel.
> 
> I don't like that.
> 
> Parsing Received: and adding the information to SMTP-level arguments is 
> bad, because it adds complexity: If Received: can be parsed, the 
> recipient MTA/Sieve/MUA can do it and there's no need for RCPT TO 
> arguments in the first place.
> 

OK.  That's simpler.  I thought more about trace= and I think
it's a half solution: you still don't know who asserted what in
the trace= field.  I'm fine with solicit= meaning it contains only
the Solicitation header keywords (and contains all of them,
not just a subset).

> I'd say that ONLY the first MTA may add SOLICIT=, and ONLY based on the 
> Solicitation field and/or the MUA's SMTP/SUBMIT SOLICIT= argument. 
> Other MTAs may not: if there are Received fields at all, neither they 
> nor Solicitation may be parsed to construct SOLICIT=.

Hmmm ... I'd simplify that one more time.  You need to be able to
grab the solicitation header in case there are non-supporting
mtas in the middle.

How about this: 

If the mua says SOLICIT=, there MUST be a Solicitation: header.
The keywords should match.

If you receive SOLICIT= using this extension, you must pass it
along intact if the next hop supports this extension.

If you receive a message from an mta that is not supporting this
extension, you MAY (not must), for operational reasons,
grab the keywords (all of them) from the Solicitation: header.

That last one is in the grey area, but I think it meets the
goals of message transport and I believe does not violate
2821 as long as we're clear on the requirement to continue
relaying.  Ned and/or John can certainly chime in.  IMHO,
this requirement is particularly important during the period 
when the extension is starting to be implemented.

> > I've assumed that the only time a message is going to be rejected 
> > based on a solicitation class keyword is by an MTA which exercises a 
> > policy under the *direct control of the recipient.* The decision to 
> > accept or reject mail is well outside of the scope of this standard. 
> > I thought that was clear, but I'd be happy to hit that point with a 
> > hammer a few more times.
> 
> This sounds good, but IMO it also implies that relays have no business 
> influencing this decision with TRACE= information of any kind.
> 

Yes.  Relays MUST simply relay messages.  Only when a message
has reached an MTA under the *direct control of the recipient*
can other actions be taken.  Is everybody happy with that
phrase before I start sprinkling it throughout the draft?  :)

Regards, 

Carl

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Date: Wed, 28 Jan 2004 09:37:57 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Carl Malamud" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Tuesday, January 27, 2004 8:40 PM
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed
Standard

>
> > 1. By far the biggest problem I have with this proposal is the idea that
> >
> > solicitation classes that are assigned by intermediaries can then be
> > copied into the SOLICIT= field of the envelope and/or used as the basis
> > for failure to deliver the mail.
>
> OK, I'm convinced.  In the message, you have a set of received headers
> and a solicitation header: you know who made what assertions.  How
> about doing this:
>
> 1. The SOLICIT= parameter in the envelope MUST contain all solicitation
> class keywords found in the Solicitation header.
>
> 2. An optional TRACE= parameter MAY contain additional solicitation
> class keywords found in received: headers.
>
> I'm fine with that, but would like to hear from some others how they
> feel.
>

I agree with Keith from the basic fundamental standpoint the (SMTP) passthru
process should not be in the mail interpretation or mail alterating game.
What about Gateways?  It will fall in the same category of recording the
Return-Path: header.  This will force the router design to begin reading the
header to prepare SMTP envelope transactions.

I now believe the proposal is too complicated for what it is trying to
achieve.

Having the client identify the intent of the mail at the smtp protocol is
highly desirable and assist in the CAN-SPAM mandate. So this part of the
concept is good but I am now leaning to believe that a SUBJ: concept
achieves the same result and it is alot simpler in design and also has
additional benefits outside the scope of this proposal.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: Arnt Gulbrandsen <[email-masked]>
To: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Date: Wed, 28 Jan 2004 14:04:42 +0100
Sender: [email-masked]

Carl Malamud writes:
>>  1. By far the biggest problem I have with this proposal is the idea 
>>  that solicitation classes that are assigned by intermediaries can 
>>  then be copied into the SOLICIT= field of the envelope and/or used 
>>  as the basis for failure to deliver the mail.
>
> OK, I'm convinced. In the message, you have a set of received headers 
> and a solicitation header: you know who made what assertions. How 
> about doing this:
>
> 1. The SOLICIT= parameter in the envelope MUST contain all 
> solicitation class keywords found in the Solicitation header.
>
> 2. An optional TRACE= parameter MAY contain additional solicitation 
> class keywords found in received: headers.
>
> I'm fine with that, but would like to hear from some others how they feel.

I don't like that.

Parsing Received: and adding the information to SMTP-level arguments is 
bad, because it adds complexity: If Received: can be parsed, the 
recipient MTA/Sieve/MUA can do it and there's no need for RCPT TO 
arguments in the first place.

I'd say that ONLY the first MTA may add SOLICIT=, and ONLY based on the 
Solicitation field and/or the MUA's SMTP/SUBMIT SOLICIT= argument. 
Other MTAs may not: if there are Received fields at all, neither they 
nor Solicitation may be parsed to construct SOLICIT=.

TRACE= is a way of saying "while relaying, I looked at this message and 
inferred that blah blah" which seems to open the door to activist 
relays. IMO opening that door is bad and would need strong 
justification. (In principle I think activist relays are okay as local 
policy. Local policy is beyond internet standards.)

> There is one thing that I do think might help address your concern. 
> I've assumed that the only time a message is going to be rejected 
> based on a solicitation class keyword is by an MTA which exercises a 
> policy under the *direct control of the recipient.* The decision to 
> accept or reject mail is well outside of the scope of this standard. 
> I thought that was clear, but I'd be happy to hit that point with a 
> hammer a few more times.

This sounds good, but IMO it also implies that relays have no business 
influencing this decision with TRACE= information of any kind.

Arnt

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: "Subject" in the envelope (was: Re: Comments requested on draft-malamud-no-soliciting-04.txt)
Date: Wed, 28 Jan 2004 03:20:34 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Monday, January 26, 2004 11:12 PM
Subject: Re: "Subject" in the envelope (was: Re: Comments requested on
draft-malamud-no-soliciting-04.txt)

> > Incidentally, this delayed MAIL FROM validation concept is
> > suggested in '2821'.  The same can written about SUBJ.
>
> Yes, I know.  I wrote it :-)

Yes, I know. :-)

> You might find it interesting to lurk on the "[email-masked]"
> list for a while.

I just subscribed. I'm catching up with the archive.

> >From my perspective, the internationalization issues are more
> important, long-term, than the spam ones.   Internationalization
> will be with us forever, or at least as long as people use
> different languages and scripts.  I think that by, say, a decade
> from now, we either will have the spam situation under control
> or it will have killed email as we know it --regardless of
> protocols-- as a useful tool.

I see your point.  I don't have much to say about internationalization at
the moment, but I intend to put more time into it.

> I am not the only person who has
> reached that conclusion although I have no idea whether we
> constitute any sort of majority or consensus.  You may disagree,
> but that conclusion is the main reason why changes that are
> helpful to both spam-fighting and internationalization tend to
> get more support than one that attacks only one of them
> (especially only spam).

I don't disagree with you if you feel internationalization is important.  My
only basic feeling is that I have a fundamental (and also long standing
ethical) design approach that a data transport system is just that - a
transport system and it has no business in the "data interpretation" game,
especially for passthru system.  With the exception of network control
lines, the only thing I feel SMTP should be dealing with is client/server
access control concepts in order to established an authorized or acceptable
transaction.  The AVS (AntiVirus/Spam) problem has highlighted this very
clearly.   So although I would think it is two separate issues,  I don't
disagree that both should be part of a new solution.  I trust your judgment
on the internationalization issues.

You brought up the excellent point about the SUBJ line idea and
international string characters.  That has to be considered in the design
for this idea, if it is implemented.

> I've often found it helpful to separate the question of "what
> information do we to need to make decisions" from "what do we
> want to log".  I clearly haven't worked through a complete
> analysis, but, for this situation and your proposal, my quick
> reaction is that I'd want to have SUBJ/  MAIL/ RCPT / Trace /
> DATA because:

Right. I agree.   I don't think having this particular order will effect
specific implementation designs by vendors to offer auditing/tracing and or
different delay validation logics.

> > We learned from our CBV work that it works best to delay the
> > MAIL FROM validation until the RCPT is provided.   If RCPT is
> > going to refused,  there is no need to perform the CBV.  This
> > was an overwhelming major improvement since atleast 60-70%
> > recipients attempts on our site are rejected.   In any case, I
> > agree. SUBJ should probably precede the other commands and
> > implementations can chose when to validate it.
>
> You presumably then do the CBV on the first otherwise-successful
> RCPT command and not before or after that, correct?

That is correct John.  It is only done once when the first "RCPT TO:" is
determined to be acceptable address that would allow the session to
continue.   When CBV is called and it is returns a rejection response to
SMTP, then SMTP will reset the state machine and force the client back to
MAIL FROM.  If CBV returns a positive response, it is no longer called if
there are additional RCPT commands.

> That could lead to some rather strange patterns of error replies,
> depending on the sequence of RCPT commands, but it might well be worth it.

Good observation.   What I have found is that the end result is the same
whether sessions are done with multiple RCPT or individually (when true
rejections is detected).  Nothing we can do about a CBV passing the test. I
could go into detail with real SMTP log examples illustrating this if you
like to see this.

The only specific design implementation point I can make is that if a CBV
rejection occurs when called at RCPT, the SMTP server resets the state
machine forcing the client back to the MAIL FROM state.   This has helped
stop spammers from continuing to issue new RCPT commands. This helped to
also reduce any RCPT user validation overhead.

Off the top of my head, without rechecking, I seem to recall a  thought
process where this might be violating the specification.   Can you verify
this for me?

Forget about CBV.   What if a client was issuing multiple RCPT commands and
for whatever reason, a rejection threshold was reached?   What can the SMTP
server do at this point?

Thanks

PS:  I am going to go ahead and write a draft for SUBJ.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CERT Advisory CA-2004-02 Email-borne Viruses
Date: Wed, 28 Jan 2004 01:28:21 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "IETF-SMTP" <[email-masked]>
Sent: Tuesday, January 27, 2004 10:57 PM
Subject: Re: CERT Advisory CA-2004-02 Email-borne Viruses

>
> Well, they rely on a combination of human intervention -- in that they
> require humans to actually "open" the attachment -- and violations of
> the MIME specification by the recipient's MUA.
>
> A big part of the problem is that when the message attachment is
> opened, the MUA then executes the content, despite the admonition of
> the MIME specifications that
>
> (a) an MUA should not allow the sender of a message to specify what
> action the recipient takes to display the attachment (which the sender
> effectively does by specifying the filename suffix) and
>
> (b) for types not known to be safe the MOST an MUA should do is to
> offer to save the attachment in a file.
>

Good points Keith.   Of course,  promiscuous behavior can get one really
dirty :-)  Unknown attachments or downloads should not be trusted and open
in any way.

However,  I was hoping to reference this not to highlight the end result,
but the distribution process the SORBIG generation of email viruses have
exploited.  The new email viruses in this advisory fall in the same
distribution category based on the exploitation of the SMTP functional
specification made famous by SORBIG.  This will continue until the real
problem is recognized, addressed and solved at the SMTP level.   Developers
will have little to offer as long as the specs relaxed security and
resistance to enforcement is written in stone.

The email-virus distribution logic depends on 3 things:

1) Spoofing other people's valid or partially valid address/domain for
distribution,
2) Spoofing YOUR valid or partially valid address/domain for distribution,
3) Exploiting Anti-Virus Software automated SMTP DSN/error reporting to
expand the distribution.

This is where the focus needs to be as it hits the heart and soul of the
problem we are facing today with the SMTP functional specification.

In short, they work by using valid or partially valid MAIL FROM and valid
RCPT addresses.  A group of the mail may or may not make it to the user.
Those systems with AVS software will help the virus distributing by bouncing
the mail back to the return paths.

During SORBIG, depending on the AVS software,  one of the problems was the
bounce loops created.  So the better AVS software has learned not to include
the attachment in the bounce and/or also learned how to use a proper NULL
Mail From.  This round of email-viruses seem to indicate the AVS software
have learned to better handle this.  The attachments are not included.

FWIW,  Our CBV has addressed a majority of these emails the past few day,
but of course, not all.   i.e, a spoofed return path with a invalid user
part but valid domain part.  The CBV will catch all of these.   However,  of
course, it will not catch a complete valid email return path address.     In
our case,  those getting by the CBV were caught with our  AV mail filters.
The result:  zero (0) percent penetration.

(Please note, by no means,  is the above statement an attempt to convince
you or to solicit endorsement. I am just reporting an observed result of a
CBV helping to curb the current set of email SORBIG-like viruses).

Anyway, starting brand new,  throwing out the door a CBV concept,  what can
be done to address the above distribution exploitation?

I think with current SMTP specifications, I can only see end-point
solutions,  as it is today by depending on AVS software.  That is not bad
and I don't think we can completely eliminate end-point mail filtering
technologies.  We will always need it or rather the admins and end users
will always need to have it around.

For a SMTP revamp,  what can be done at the SMTP level, if any, to minimize
or prevent this distribution exploitation?

Given the information a client provides at the SMTP level,  regardless of
how its done,  it boils down to validating the  sender machine and/or return
path.   How can we do this in a highly optimize fashion?

I got several ideas I am working on, but darn it,  it keeps coming back to
the functional specs.

How can you work on an idea of validating a return path when the darn specs
allows for it to be non-verifiable?     I mean, just consider the IMC.ORG
MAILSERV file request system.   It uses an unverifiable
[email-masked].   How  can anyone  can any solution done when the
IETF illustrates by example that the return path is not verifiable?   Why
should spammers or anyone for that matter follow 'guidelines' when the IETF
itself can not?

Don't know about you, but there is something wrong with this picture. The
IETF should be the #1 system in the world showing how the protocols should
work as it was intended to work.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Tue, 27 Jan 2004 20:38:45 -0800 (PST)
From: [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> If we're going to start standardizing spam countermeasures I'd like to
> see IESG or the Apps area adopt a strategy for dealing with the variety
> of proposals I suspect we're going to see, as various groups seek to
> gain legitimacy for their work.  For instance:

> 1.  Since I doubt we'll be able to decide exactly which set of
> countermeasures should be widely implemented, my guess is that we'll
> end up publishing several competing proposals at Experimental and/or
> Proposed and re-evaluating them after a time in light of operational
> experience and market acceptance.   If we are going to do that we should
> publicly say so up front, so that we have some semblance of uniformity
> and we aren't as likely to be accused of favoritism.

I think this is the course we're on, like it or not.

> 2.  Some spam countermeasures are "riskier" than others, in that they
> use less-than-reliable criteria for classifying mail as spam, or they
> adversely affect mail reliability for other reasons, or they change
> SMTP or the mail format in inappropriate ways.  If we are going to
> publish risky proposals I'd strongly prefer that we publish them as
> Experimental rather than Proposed.  Proposed standards should "do no
> harm".

Agreed. However, I don't think this proposal qualifies as particularly
risky.

> However there still might be some value in publishing risky proposals
> as Experimental -

> First, to invite review of those proposals by experts within IETF and
> where feasible, to encourage modification of those proposals to minimize
> their potential for harm

> Second, to encourage modification of those proposals so that conflicts
> between them are minimized;

> Third, to provide the opportunity to clearly label which proposals are
> seen to be risky and for what reasons

This reason resonates the most with me. I've seen the notion of "it is
better not to publish stuff with issues" backfire far too often now.

> Fourth, to encourage better collection of experience with
> implementations of such proposals  (since people are already widely
> using risky countermeasures, we can't expect to stop them - but we might
> as well try to learn from it)

Yep.

> Layer 2: this proposal specifically

> 1. The basic idea seems good and "mostly harmless", but I think it needs
> substantial revision before being adopted.  With appropriate changes,
> I'd recommend it for Proposed PROVIDED there's a pre-announced policy
> on how we're going to deal with similar proposals.  I don't think we
> should be appearing to endorse this specific approach over other
> approaches at this time.

It isn't clear to me what form such an announcment should take. I suppose
the IESG could include something about it in the approval notice.

> 2. [section 2] Nit: to me it seems confusing for solicitation-keywords
> to be a single ehlo-param (in RFC 1869/2821 parlance) and for the
> keywords to be comma-separated.  I'd like to see us keep EHLO responses
> fairly simple and uniform in format.  Have we had comma-separated
> keywords in other EHLO responses?

No, but we do have at least one ELHO parameter that accepts a
keyword list: AUTH. Spaces were used in this case, however, I cannot
say I forsee a major problem with using commas here. OTOH, I have no
problem with switching to spaces for consistency with AUTH.

> I guess the idea is to keep one syntax for "solicitation-keywords"
> that can be used in both EHLO response and MAIL FROM command.  That
> might be better than having two separate grammars.

Yes, that's the tradeoff, and its why I didn't recommend a change
to the EHLO syntax in my review. I can argue this one either way.

> 3. [section 2] Nit: it would be better if the entire grammar for
> the EHLO response were in one place, rather than spread over
> multiple sections.

A collected grammar at the end is better still, and I believe Carl
has stated that he plans to do this.

> 4. [section 2.1] Nit: the last paragraph seems out of place;
> I think it should go in an acknowledgements section near the
> end of the document.

Alterately, the convention we used in the MIME specification of having
HISTORICAL NOTES could be used instead.

> 5. [section 2.2] I understand the need to define some initial
> solicitation classes but I'd like to see us avoid endorsing
> these.  The MAPS-UBE class in particular, while it makes an
> interesting topic for discussion, doesn't strike me as something
> that we should endorse by making it a built-in part of a standards-
> track protocol.  2.2.1 does say "for illustrative purposes only"
> but similar disclaimers in other standards have been disregarded.

Yes, this one made me nervous as well.

> More generally, what I'd like to see is a separation between the
> specification of the protocol engine (i.e. the grammar) and the
> specification of solicitation classes.   (I realize that we've
> traditionally included some keywords in grammars for other protocol
> specifications.  I think we need to stop doing this, as it makes
> the grammars more complex and less useful for both protocol
> implementation and constructing protocol verifiers.)

Sounds reasonable to me.

> So the protocol engine is what should be standardized, and we should
> have some initial solicitation classes registered that are not part of
> the standard (they could go in an appendix to the document labelled
> "initial solicitation class definitions" that says "not part of the
> standard").

Good.

> It would be okay with me to use some of the initially defined classes
> in examples.

> Finally I think this is one of those cases where x- fields might not
> be a good idea - at least, I'd like to encourage public review of
> new solicitation classes.

I was wondering what you and others would say about this... I tend to
agree that this is a place where X- could be harmful.

> The grammar should be rewritten to reflect these various changes.  For
> instance:

>      Solicitation-keywords = registered-word 0*("," registered-word)

>      registered-word = ALPHA 0*(wordchar)
>                                   ; registered-word(s) are registered
>                                   ; with the IANA
>      wordchar = ("-" / "_" / ":" / ALPHA / DIGIT)

Yes, this is cleaner.

> 6. [section 2.3]  Is it possible that the recipient doesn't want to
> disclose the reasons for rejecting the mail?  Perhaps there should
> be some generic indication that the message was rejected due to
> policy rather than because the recipient address was not valid.
> (550 is the right code but is probably overused.)  Or maybe this
> proposal should just require ENHANCEDSTATUSCODES as 5.7.1 is a
> clear indication that this is a policy violation without saying
> exactly what policy is violated.

Now that you bring it up, some mention of the enhanced status code
to use is probably in order here.

> If nothing else this should make clear whether the policy indication
> is in some machine-readable format (is it part of the protocol?
> if so, what's the precise syntax?)  or is meant to be human readable.

> Is it conceivable that a recipient might be willing to accept some kinds
> of solicitation but impose various limitations or quotas on that kind
> of mail?  In this case a 4xx error code and 4.7.1 status code might be
> appropriate to indicate "go away, come back later".

How about stating that an implementation MAY impose limits such as
message size restrictions based on solicitation classes, and that when
such limits are exceed they SHOULD be reported using whatever status
code is appropriate for that limit?

> 7. [section 2.6] I have strong objections to various parts of this
> section:

> - Most of this proposal is about giving the sender a way to assert
> that a message is a solicitation, and that seems to me to be
> "mostly harmless".  Giving intermediaries a way to make similar
> assertions is "risky" and I don't think such a practice should be
> standardized.

> For example: a sender can assert (rightly or wrongly) that an
> advertisement is solicited by a recipient and be in a position to know
> whether that assertion is true; an intermediary cannot be expected to
> know whether such an assertion is true.

> - Letting intermediaries' assertions get confused with the sender's
> assertion (as provided for in section 2.x) is even worse, and I'd regard
> that as something that should not even be published as Experimental.

> - Even accepting that it's useful for intermediaries to make assertions
> about the solicitation classes of a message, it's not at all clear
> that the classes that can be asserted by an intermediary are similar
> to those that can be asserted by a sender.  An intermediary can
> make assertions about content (e.g. "this message contains a GIF image")
> but is not in a good position to make assertions about intent.

> - Even accepting that it's useful for intermediaries to make assertions
> about the solicitation classes of a message, I don't think the Received
> field is a good place to do that, and I don't like the idea of storing
> protocol elements in comments or of having MTAs read and trust the
> contents of comments.   MTAs should as much as possible treat message
> content as opaque.

> I strongly recommend that this document NOT provide a way for
> intermediaries to make assertions about solicitation classes,
> as that seems orthogonal to the main purpose of this document,
> and it seems to make it much riskier.

The risk has to be weighed against the desireability of being able to capture
such assessments and being able to associate then with a specific agent. This
is another one I can argue either way.

> Even accepting that it is useful for intermediaries to make
> assertions about solicitation classes, the protocol for doing this
> should be defined in a separate document, and those assertions should be
> kept cleanly separated from assertions made by the message originator,
> and there should probably be different classes for those indications.

> 8. [section 2.7] The wording is muddy.  It appears to apply only
> to SYSTEM-WIDE, and that doesn't seem right.

> IMHO, anytime a server asserts NO-SOLICITING in EHLO, this means:

> - if NO-SOLICITING SYSTEM-WIDE is asserted and a set of prohibited
> solicitation classes is included, client SHOULD NOT attempt to relay
> mail for which the SOLICIT=  MAIL FROM parameter includes a class that
> is prohibited in the EHLO response.

>   (however if the client does so it's still the server's responsibility
>   to reject it)

> - regardless of whether a set of solicitation classes is included
>   in the EHLO response, client MUST convey SOLICIT= faithfully (i.e.
>   as received from the previous hop) in MAIL FROM for each message
>   relayed to a server that supports NO-SOLICITING.

Seems reasonable to me.

> 9.  Actually I suggest getting rid of the SYSTEM-WIDE and PER-RECIPIENT
> keywords.  Here's why:

> - it's useful to allow a combination of SYSTEM-WIDE and PER-RECIPIENT
>   policies, rather than to force one or the other.  The SYSTEM-WIDE
>   policies would apply to all recipients supported by that server,
>   but individual recipients could have more restrictive policies.

Interesting notion. I think I like it. So you'd have a list of "globally
unacceptable" keywords in the EHLO announcment but allow per-recipient
rejection on the basis of the SOLICIT= MAIL FROM parameter? Yes, this
definitely is an improvement.

> - it's useful for an MTA to have the ability to relay SOLICIT=
> indications even if it doesn't actually impose system-wide or
> per-recipient restrictions.  Yes, it could claim PER-RECIPIENT
> and then never impose any restrictions, but that seems misleading.

Of course. I missed this issue entirely; sorry.

> I'd also suggest changing the name of the EHLO keyword because
> supporting relaying of SOLICIT= doesn't imply that the MTA has any
> restrictions on soliciting.   So I suggest:

> - rename the EHLO keyword something like "RESTRICT"
> - allow the EHLO keyword to have system-wide options.  e.g.

>    RESTRICT means that the SOLICIT= keyword is supported but that
>    no system-wide restrictions are advertised, and

>    RESTRICT=class1,class2,... means that the SOLICIT= keyword is
>    supported and that class1,class2,... are prohibited on a
>    system-wide basis.

Careful here - there is no equals in the syntax here. (I'm sensitive to this
issue because Microsoft botched its handling of AUTH and now servers may have
to advertise AUTH=FOO in addition to AUTH FOO.)

> 10. Somewhere some text like the following seems appropriate:

> Note that the system-wide option, while useful, is of marginal
> value.  Many MTAs want to be able to accept mail for multiple domains
> that may have different policies, and any large user community will
> probably need to have different policies.  Typically there are some
> people in an enterprise who _need_ to be able to receive advertisements-
> e.g. people who do procurement and who naturally need to know what goods
> are available at what prices from which sources.

Honestly, I don't think it is all that marginal. There are lots of fairly
specialized SMTP servers for which the system-wide option makes considerable
sense.

> 11. [section 2.8]  The ability to relay SOLICIT= SHOULD be enabled by
> default. However the default configuration SHOULD NOT impose any
> system-wide or per-recipient policy.

Agreed.

> An MTA that advertises the EHLO keyword MUST relay SOLICIT= faithfully
> to any next-hop MTA that advertises the extension in EHLO.

Agreed.

> An MTA that advertises the EHLO keyword MAY implement the ability to
> advertise a system-wide policy in EHLO.  If such a policy is advertised
> it MUST be enforced by rejecting messages for which MAIL FROM asserts
> a solicitation class that conflicts with that policy.  (It MAY also
> be enforced by other mechanisms, e.g. looking at the Solicitation:
> field and rejecting messages for which that field conflicts.)

For completeness, sure.

> An MTA MAY also implement a system-wide policy without advertising
> all of that policy in EHLO.

Why not?

> An MTA that advertises the EHLO keyword MAY implement per-recipient
> policies for zero, some, or all of its recipients.

Sure.

> 12. [section 3]  I think this belongs in an appendix, as it seems
> to be non-normative.

Seems reasonable to me.

> 13. It is fairly obvious that sites and recipients will want policies
> based on reliable characteristics of the message other than a
> solicitation class advertised by the sender.  e.g. I want to be able to
> reject application/ms-word documents.  And then we also have things like
> the content-negotiation stuff that the fax people want.  Given this,
> what is the potential for conflicts between these various policy
> mechanisms?  And should we try to sort them out now or wait until later
> when we have more experience with them?

I think waiting on this would be a mistake for a variety of (probably fairly
obvious reasons).

				Ned

Cc: Keith Moore <[email-masked]>, "IETF-SMTP" <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: CERT Advisory CA-2004-02 Email-borne Viruses
Date: Tue, 27 Jan 2004 22:57:14 -0500
To: "Hector Santos" <[email-masked]>
Sender: [email-masked]

>  Unsolicited email messages containing attachments are sent
>    to  unsuspecting  recipients.  They  may  contain  a return 
> address, a
>    provocative  envelope,  or something else that encourages its 
> receiver
>    to  open  it.  This technique is called social engineering. Because 
> we
>    are  trusting  and curious, social engineering is often effective. 
> The
>    widespread  impact  of  these  latest  viruses,  which  rely  on 
> human
>    intervention  to  spread,  demonstrates  the  effectiveness  of 
> social
>    engineering.

Well, they rely on a combination of human intervention -- in that they 
require humans to actually "open" the attachment -- and violations of 
the MIME specification by the recipient's MUA.

A big part of the problem is that when the message attachment is 
opened, the MUA then executes the content, despite the admonition of 
the MIME specifications that

(a) an MUA should not allow the sender of a message to specify what 
action the recipient takes to display the attachment (which the sender 
effectively does by specifying the filename suffix) and

(b) for types not known to be safe the MOST an MUA should do is to 
offer to save the attachment in a file.

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: CERT Advisory CA-2004-02 Email-borne Viruses
Date: Tue, 27 Jan 2004 21:44:44 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

 This document is available from:
   http://www.cert.org/advisories/CA-2004-02.html

I. Description

   Over  the  past  week,  we  have  seen  two more mass-mailing viruses,
   W32/Bagle  and  W32/Novarg,  impact a significant number of home users
   and  sites.  The technology used in these viruses is not significantly
   different  from  prior  mass-mailing  viruses  such  as  W32/Sobig and
   W32/Mimail. Unsolicited email messages containing attachments are sent
   to  unsuspecting  recipients.  They  may  contain  a return address, a
   provocative  envelope,  or something else that encourages its receiver
   to  open  it.  This technique is called social engineering. Because we
   are  trusting  and curious, social engineering is often effective. The
   widespread  impact  of  these  latest  viruses,  which  rely  on human
   intervention  to  spread,  demonstrates  the  effectiveness  of social
   engineering.

....

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Keith Moore <[email-masked]>
Date: Tue, 27 Jan 2004 17:40:40 -0800 (PST)
CC: [email-masked], [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

> 1. By far the biggest problem I have with this proposal is the idea that
> 
> solicitation classes that are assigned by intermediaries can then be
> copied into the SOLICIT= field of the envelope and/or used as the basis
> for failure to deliver the mail.  

OK, I'm convinced.  In the message, you have a set of received headers
and a solicitation header: you know who made what assertions.  How
about doing this:

1. The SOLICIT= parameter in the envelope MUST contain all solicitation
class keywords found in the Solicitation header.

2. An optional TRACE= parameter MAY contain additional solicitation
class keywords found in received: headers.

I'm fine with that, but would like to hear from some others how they
feel.

> Well, there are really two issues here.  The first and most important,
> as I mentioned above, is that the current proposal allows
> sender-supplied and intermediary-supplied information to go in the same
> field in the envelope.  The second is that both mechanisms (sender
> labelling and intermediary labelling) are being described in the same
> document, when the two have very different degrees of reliability and
> applicability.  The document is our quantum unit of standardization - it
> is awkward for us to say "this part of the document is proposed standard
> and recommended, while this other part is experimental and dangerous 
> to interoperability".  
> 
> Also, the bulk of this document is discussing a mechanism for senders
> to label their content, and for MTAs to convey that labelling and to
> convey recipient and/or site policies for accepting that content.
> The mechanism for intermediaries to label content, while it's certainly
> a related topic, doesn't currently get the amount of attention it
> deserves.
> 

I think it would be dangerous to give separate mechanisms.  As you
said later in your note, a clueful MTA wouldn't want to use the same
keywords.  But, rather than us making those decisions, I'd rather
provide a simple registry mechanism and then let users decide
what the words mean.  All we're saying in this extension is that
the sender or an MTA asserts a series of keywords.  We simply deliver 
the mail.  

There is one thing that I do think might help address your
concern.  I've assumed that the only time a message is
going to be rejected based on a solicitation class keyword is
by an MTA which exercises a policy under the *direct control of the 
recipient.*  The decision to accept or reject mail is well outside 
of the scope of this standard.  I thought that was clear, but
I'd be happy to hit that point with a hammer a few more
times.

>  
> > 4. You indicated that we might want to do both
> > SYSTEM-WIDE *and* PER-RECIPIENT (e.g., more
> > restrictive policies for one user).  I thought
> > about that a lot and was worried that implementation
> > becomes significantly harder.  My goal with
> > system-wide was to provide a *really* easy entry
> > point.  I could hack sendmail to do system-wide: there
> > is no way I could program per-recipient myself.
> 
> Maybe I wasn't clear.  I want the protocol definition to support the 
> ability of a single MTA to impose both system-wide and per-recipient
> policies if it wants to do that.  

I'm fine with that.  System-wide defines a base class and may be
optionally supplemented by additional per-recipient policies.

Again, would like to hear if anybody else has strong opinions
on this.

> I also want the protocol definition
> to allow a conforming MTA to request SOLICIT parameters from clients
> and convey SOLICIT parameters to receivers even if it doesn't impose
> any restrictions on mail relayed through that MTA.
> 

That's already in there.  This proposal is about the
conveying of keywords *not* the decision to accept or
reject messages.  That is, potentially, the subject
of other standards or laws, but this proposal is a
mechanism for the message sender and others involved
in the delivery of the message to assert keywords,
and for the recipient to assert keywords.  These
words are all drawn from a registered list.  The
decision to 250 or 550 or to attempt delivery or
not is not what we're specifying here.

If we split the source/intermediate keywords in the
envelope, allow system-wide+per-recipient, and 
make it *really* clear that just because a keyword
matches, we're not telling you what to do, have
I addressed your concerns?  (This is in addition
to the cleanup of grammer, deletion of fluff,
etc... we talked about yesterday.)

Regards,

Carl

Date: Tue, 27 Jan 2004 10:45:44 -0500
To: "Hector Santos" <[email-masked]>, <[email-masked]>, <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

Hello Hector,

At 21:35 04/01/26 -0500, Hector Santos wrote:

>What I did not get 100% clear about SPF, is whether it works from a central
>authority (database) concept.  Its documentation seems to indicate both (a
>central and your own).  The author should clarify this more.

Please tell him directly. I don't have any direct contacts.

>In any case, in my research and testing of the various DNS lookup proposals,
>the only "current" benefit I see from all the anti-spam DNS based lookup
>proposals is checking spoofs against your own domains.

That's the initial benefit, as I mentioned.

>If you begin to
>check other domains, the DNS overhead skyrockets.   It was pointed out to me
>by the DMP author that this is a function of your DNS setup. I am not too
>sure about this. But I don't pretend to be an DNS expect.  All I know is
>that looking up an unknown domain (which is what a majority of the spammers
>are) yields a long initial lookup delay.

I'm no DNS expert either. But giving the spammers some delay may be a
nice side effect :-).

Regards,   Martin.

Date: Tue, 27 Jan 2004 13:44:49 -0500
From: Keith Moore <[email-masked]>
To: Carl Malamud <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

okay, I'll try to order this by priority...

1. By far the biggest problem I have with this proposal is the idea that

solicitation classes that are assigned by intermediaries can then be
copied into the SOLICIT= field of the envelope and/or used as the basis
for failure to deliver the mail.  This is a showstopper as far as I'm 
concerned.  Allowing the sender to label a message as an advertisement
is clearly a Good Thing.  Allowing an intermediary to label a message
is probably also useful.  Combining the two into a single indication is
definitely a Bad Idea that will lead to even more failures of the mail 
transport that are difficult to track down and fix.  An indication of
sender intent by an intermediary is almost inherently unreliable, and
a lot of the concerns expressed in RFC 3238 seem at least as applicable 
to this aspect of your proposal.

The simplest way to fix this is to forbid MTAs and intermediaries from
copying classification information from the Received field (or wherever
else it ends up being stored) to the message envelope.  Note that this
doesn't prevent filtering of messages based on classification
information supplied by intermediaries and stored in the message
payload; what it accomplishes is to keep sender-supplied information
separate from intermediary-supplied information.

> 1. I agree the grammar is confusing being spread out.  I'll
> keep it as is because it leads the reader through the pieces,
> but then collect those pieces in an appendix.

thanks.

> 2. I picked 3 sample classes because I needed something for
> examples.  Let me move those to the appendix, mark them
> clearly as "not part of the standard," modify the grammar,
> then use them in the text as examples.  Does that work for you?

yes, thanks.

> 3. You had some good comments on enhanced status codes.  I've
> now received 3 suggestions on the answer (from you originally,
> then Eric Allman, then Ned).  I think that section (2.3),
> should be looser: point out 3 possible error codes, say
> that this is a topic of other standards.

5.7.1 will nearly always be the right code.  SHOULD should be about
right, and I'd rather see this specified here than in other standards.

My main concern in my last message about responses was that temporary
failure codes might also be reasonable in some circumstances.  This
affects both response codes and  enhanced status codes. 

> 1. You worried a bit about the comma-separated list of
> keywords.  As you guessed, the reason was to keep a
> uniform syntax in all four places: EHLO, MAIL FROM and
> answers, Solicitation: header, and received: header.
> Comma-separated with no spaces and a limited character
> set seemed like the easiest solution.

I'm okay with this.

> 2. You had a strong objection to the trace fields for
> two reasons: the basic idea of an intermediate
> doing things and the mechanism of the comment field.
> I agree intermediates should leave the message alone.
> Today, most spam filters inject new, non X- headers
> into messages.  It seemed to me that 2821 and 2822
> would both be happier if that gunk was moved up to the one
> line those folks were supposed to touch.  The use
> of the comment mechanism was Ned's request: he says
> we've burned that bridge before and that was the
> best place for this information.

Mumble.  Adding of fields by intermediaries is such a well-entrenched
practice that I don't see any hope of getting intermediaries to keep the
headers pristene.  I'll be happy if we can get intermediaries to stop 
modifying existing header fields and if we can define future fields in
such a way that it is clear as to who said what.

OTOH, one nice thing about using Received, is that it makes it clear
_which_ intermediary added the classification information.  And since
a big part of my concern about labelling of intent is that
intermediaries cannot do so reliably, I really like the idea that we
should keep track of which intermediary said what about the message.  If
the information were put in a separate header I'd still want the name of
the intermediary (and where it fit into the signal path) to be included.

So while I'm still not entirely comfortable with using Received for this
purpose, I now see more benefits to doing so than I originally did.

> 3. You had even stronger objections to having
> message senders mixed in with intermediates in one document.

Well, there are really two issues here.  The first and most important,
as I mentioned above, is that the current proposal allows
sender-supplied and intermediary-supplied information to go in the same
field in the envelope.  The second is that both mechanisms (sender
labelling and intermediary labelling) are being described in the same
document, when the two have very different degrees of reliability and
applicability.  The document is our quantum unit of standardization - it
is awkward for us to say "this part of the document is proposed standard
and recommended, while this other part is experimental and dangerous 
to interoperability".  

Also, the bulk of this document is discussing a mechanism for senders
to label their content, and for MTAs to convey that labelling and to
convey recipient and/or site policies for accepting that content.
The mechanism for intermediaries to label content, while it's certainly
a related topic, doesn't currently get the amount of attention it
deserves.

So I still think that a separate document for intermediary-labelling
would be far preferable.  It could certainly refer to the
sender-labelling document as needed.  A fallback position would be to
place the intermediary-labelling in an appendix, but I really don't
think it belongs in the same document at all.

> It seems to me that in the real world, both actors
> are going to be conveying information that is
> similar, and I've tried to come up with a uniform
> mechanism that allows the user to ultimately sort
> out who they want to trust. 

I agree that there might be some merit in having a common set of
solicitation classes for sender-supplied and intermediary-supplied
information, though I suspect that if the intermediaries are honest
there won't be much overlap between sender-supplied labels and
intermediary-supplied labels - because intermediaries are simply not in
a good position to judge sender intent. The same message can be either
valid or spam depending on whether the recipient has expressed an
affirmative preference to receive messages from that sender or on
that topic, and that's not information that is likely to be accessible
by an intermediary.  I might want to see ads from certain hand-picked
vendors of adult products even if I don't want to receive adult-oriented
advertisements in general.

>  Today, it would be
> hard for my MUA to correlate an "ADV" in the
> subject line, my Spambouncer headers that tell
> me this might be pornography, and the senders
> first MTA that inserted headers saying "this is
> clean." 

I don't see how your proposal helps that situation much.  We're
inherently going to have different parties with varying degrees of
trustworthiness making sometimes-conflicting assertions about a message.
The best we can do under such circumstances is to keep the various
assertions separate.

Trying to get them to use a common language might help slightly in 
that there will be a clear definition of each class. However I think
it likely that intermediaries will mis-label messages if they try to use
classes like those described in this proposal.
 
> 4. You indicated that we might want to do both
> SYSTEM-WIDE *and* PER-RECIPIENT (e.g., more
> restrictive policies for one user).  I thought
> about that a lot and was worried that implementation
> becomes significantly harder.  My goal with
> system-wide was to provide a *really* easy entry
> point.  I could hack sendmail to do system-wide: there
> is no way I could program per-recipient myself.

Maybe I wasn't clear.  I want the protocol definition to support the 
ability of a single MTA to impose both system-wide and per-recipient
policies if it wants to do that.  I also want the protocol definition
to allow a conforming MTA to request SOLICIT parameters from clients
and convey SOLICIT parameters to receivers even if it doesn't impose
any restrictions on mail relayed through that MTA.

So a conforming MTA should be able to:
a) implement system-wide policy OR
b) implement per-recipient policy OR
c) implement both a system-wide and a per-recipient policy OR
d) implement neither and just convey SOLICIT parameters on relayed mail

> Note also that people seem to have strong opinions
> about which of the two is most useful.  You
> indicated that system-wide wasn't very attractive
> to you ... others feel strongly that they would
> *only* use system-wide.  For that reason, I felt
> it best to allow both "styles" but force a choice.

I can see why system-wide is useful, at least in some corner cases.  I
wasn't suggesting that it be removed.  

However, I don't see why it is useful to force a choice.  Clearly a
site-wide policy can be implemented using the mechanism for
per-recipient policy, but (with the current proposal) only by forfeiting
the ability of the MTA to say upfront "we don't want this kind of
message".  I think it makes far more sense for the MTA to be able to say
"nobody at this site accepts ADV:ADLT messages" and then later on to say
"recipients Alice, Bob, and Carol also refuse to accept ADV messages".

> In sum, your biggest issue seems to be the
> idea of encouraging intermediate MTA's to
> look inside the message.

That seems like a misrepresentation.  It's one thing to realize that
intermediaries are going to look inside messages and that we aren't 
likely to stop them from doing so anytime soon; it's quite another to 
encourage that practice.  It's even worse to encourage intermediaries
to look inside messages for the purpose of labelling them according
to some notion of sender intent, because such labelling can never be
reliable.  And it's even worse to treat such labelling as if the 
sender had supplied it.

Without going into detail right now, a lot of the robustness and
integrity issues raised in RFC 3238 seem applicable here if this
proposal continues to support labelling of messages by intermediaries,
and they certainly apply if the proposal continues to encourage
MTAs to trust such labels for use in filtering.

-- 
Keith Moore                              http://www.cs.utk.edu/~moore/

From: Carl Malamud <[email-masked]>
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
To: Keith Moore <[email-masked]>
Date: Tue, 27 Jan 2004 06:15:11 -0800 (PST)
CC: [email-masked], [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

Hi Keith -

> 
> multiple layers of response here:

Indeed.  A very useful note.  Thanks for the careful read of
my draft.  As always, I think I agree with 95% of your
comments, though I suspect you might feel strongly about the
other 5% as well.  :)

> 
> Layer 1:  generally about standardizing spam countermeasures
> 

I agree with your comments about overall strategy (e.g., not setting
too quickly on one solution, not doing harm, making sure pieces 
don't conflict).  I think those issues are important, and that is
clearly an IESG call.  I'm fine with experimental, proposed, or
even a no-action action if that's what people want.

> 
> A lot of the spam countermeasures in use seem to me to be actually 
> harmful in that they decrease the ability of the mail system to deliver
> legitimate messages reliably.  Even if you consider that not filtering
> spam also affects mail reliability (either because it overloads mail
> systems or because it overloads recipients who might accidentally 
> delete a legitimate message embedded in a large batch of spam), some
> spam countermeasures still seem to do more harm than good.  User
> experience varies - what works well for one group of users may not work
> well for another.
> 

I agree 100% ... there were 2 motivations for this draft:

1. The fact that esmtp had no way for the receiver to indicate
that they don't want to be solicited.

2. The fact that our mail headers are being littered with
insertions by filtering software and, increasingly with obscure
markings like "ADV" required by policy makers.

My hope was to provide a very simple, friendly-to-the-architecture,
and neutral means of transporting this information.  I specifically
stayed away from authentication, consent, and many other issues
because I don't know how to solve those.

And, it may be very true that what I'm trying to accomplish fits
in a more comprehensive solution that addresses those issues,
or maybe in a less comprehensive solution.  What I do know is
that I don't see any less/more comprehensive solutions to this
particular problem on the table and, hopefully, my draft
will fit in the "can't hurt, might help" category.

For your specific comments, some are very easy suggestions that
I'll work into the draft quickly: 

1. I agree the grammar is confusing being spread out.  I'll
keep it as is because it leads the reader through the pieces,
but then collect those pieces in an appendix.

2. I picked 3 sample classes because I needed something for
examples.  Let me move those to the appendix, mark them
clearly as "not part of the standard," modify the grammar,
then use them in the text as examples.  Does that work for you?

3. You had some good comments on enhanced status codes.  I've
now received 3 suggestions on the answer (from you originally,
then Eric Allman, then Ned).  I think that section (2.3),
should be looser: point out 3 possible error codes, say
that this is a topic of other standards.

4. You pointed out some ambiguities in the relay language.
I agree that should be tightened up.

5. Section 3 can easily be deleted.  I put it in for a
different audience.  :)

Some of your other comments I don't necessarily disagree with,
but they are issues that I've wrestled over many times as
various reviewers have commented.  I'm not saying I necessarily
got it right, but let me try and explain how we got there.

1. You worried a bit about the comma-separated list of
keywords.  As you guessed, the reason was to keep a
uniform syntax in all four places: EHLO, MAIL FROM and
answers, Solicitation: header, and received: header.
Comma-separated with no spaces and a limited character
set seemed like the easiest solution.

2. You had a strong objection to the trace fields for
two reasons: the basic idea of an intermediate
doing things and the mechanism of the comment field.
I agree intermediates should leave the message alone.
Today, most spam filters inject new, non X- headers
into messages.  It seemed to me that 2821 and 2822
would both be happier if that gunk was moved up to the one
line those folks were supposed to touch.  The use
of the comment mechanism was Ned's request: he says
we've burned that bridge before and that was the
best place for this information.

3. You had even stronger objections to having
message senders mixed in with intermediates in one document.
It seems to me that in the real world, both actors
are going to be conveying information that is
similar, and I've tried to come up with a uniform
mechanism that allows the user to ultimately sort
out who they want to trust.  Today, it would be
hard for my MUA to correlate an "ADV" in the
subject line, my Spambouncer headers that tell
me this might be pornography, and the senders
first MTA that inserted headers saying "this is
clean." 

4. You indicated that we might want to do both
SYSTEM-WIDE *and* PER-RECIPIENT (e.g., more
restrictive policies for one user).  I thought
about that a lot and was worried that implementation
becomes significantly harder.  My goal with
system-wide was to provide a *really* easy entry
point.  I could hack sendmail to do system-wide: there
is no way I could program per-recipient myself.

Note also that people seem to have strong opinions
about which of the two is most useful.  You
indicated that system-wide wasn't very attractive
to you ... others feel strongly that they would
*only* use system-wide.  For that reason, I felt
it best to allow both "styles" but force a choice.

5. You mentioned more general restrictions,
e.g., no application/msword documents, and
your worry that those mechanisms might conflict
with no-soliciting.  You'll notice my draft
is really about transporting labels and does
not to deal with issues such as "are the
labels true" and "what actions should we take."

So, I agree with the potential for my
approach for bumping into more general solutions
in the future, and would only point out that
those solutions don't seem to exist yet and
that I've tried to set up no-soliciting so
that it is carefully isolated and fairly simple
so that it *could* coexist in the future.

In sum, your biggest issue seems to be the
idea of encouraging intermediate MTA's to
look inside the message.  I really think
that bridge was burned a long time ago and
if we can at least move those labels up into
a trace field, maybe they'll leave the messages
alone?  (BTW, when I envision how this will be
used, it seems to me that the only mta that
would do anything are the ones "close to
me" such as my isp and my host, but I understand
your worry about MTAs in the middle.)

I hope I've addressed some of your concerns.
Thanks again for the thoughtful analysis.

Carl

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: "Subject" in the envelope (was: Re: Comments requested on draft-malamud-no-soliciting-04.txt)
Date: Tue, 27 Jan 2004 08:50:07 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Hi John,  I have a separate message commenting on your other topics (subj).
For now, I wanted to respond to this specific question.

> Out of curiosity, how do you handle the following problem:
>
> Assume that mail is sent to someuser@host1 with a reverse-path
> of maybebogus@odd-domain, and that the relevant DNS records are
>
>    host1 IN MX 10 host2
>          IN MX 20 host3
>
> host2 is configured for CBV checking; host3, for whatever
> reason, is not.  The message reaches host3 (it is a bit bigger,
> more responsive, and better connected -- for purposes of
> simplicity, assume there are no firewalls or other middleboxes
> involved).  Host3 tries to pass it along to host2.  When host2
> attempts CBV validation, it gets a DNS timeout on the host1
> domain -- note a timeout, not an authoritative "no" answer.
> Since it is configured to not accept messages unless it can do
> the validation, it returns a 4yz code to host3, resulting in
> host3 requeuing and retrying the message, perhaps for a long
> time.   Host3 considers that behavior mildly antisocial, since
> this is not its problem.
>
> Your thoughts?

As with any scenario,  the CBV has no choice but to issue a 4xy response
with the likely situation that the DNS timeout is only temporary (it better
be) and the sender host honors the 4xy response as a signal to retry.

In this case,  the reality is that if the CBV has a DNS timeout problem then
so will host2 when it tries to do normal outbound mail operations. :-)    So
the real problem here is getting the admins to fix his DNS client/query
operations on host2.   Whatever problems host2 will experience as a send
mail machine, so will the CBV.   So as one of my customers painfully learned
and pointed our when beta testing wcSAP,  let me quote him:  "....make damn
sure you have good DNS! [sic]"

No doubt,  WCSAP has definitely become in a way,  "SMTP Compliancy Tester."
It has helped many of my customers realize many inherent issues with their
setup one way or another.   CBV is the final test of a group 4 test
performed I have lumped together called wcSAP.  From the configuration file:

EnableFilterCheck    True        ; check white/black filter list
EnableDMPCheck       True        ; perform new DNS based DMP check
EnableRBLCheck       True        ; perform RBL database check
EnableCBVCheck       True        ; perform CallBack Verification

So the customer has the power to define what checks he wants to do.

The White/Black Rules/Filter list is where I have reject/accept rules I call
DIP attributes (Domain/IP pairs).  For example for the imc mailing list:

accept if %DIP%   = mail.imc.org[208.184.76.39]
accept if %DIP%   = imc.org[208.184.76.39]
accept if %DIP%   = jck.com[209.187.148.211]  ; John Klensin (IETF-SMTP)

As you can see, I added a DIP for you to avoid the unnecessary overhead. :-)
and btw, recently I had a DIP for the site www1.imc.org site:

accept if %DIP% = www1.ietf.org[132.151.6.22]

because I was getting mail from [email-masked] which is not a
verifiable return path.  Here is one of the rejection logs for it:

-------------------------------------
state      : rcpt
cip        : 132.151.6.22
cdn        : optimus.ietf.org
from       : <[email-masked]>
rcpt       : <[email-masked]>
srvdom     : winserver.com
srvip      : 208.247.131.9
sapfilter  : pass
sapdmp     : skipped
saprbl     : testing 22.6.151.132.sbl.spamhaus.org
saprbl     : testing 22.6.151.132.list.dsbl.org
saprbl     : testing 22.6.151.132.bl.spamcop.net
sapcbv     : total mx records: 0
try domain : www1.ietf.org ip: 132.151.1.19
# connecting to 132.151.1.19
! connect error 10060
smtp code  : 452
reason     : Rejected by WCSAP CBV

Is it CBV's fault that this site is using a bad return path?  It is just
returning the truth - this site is setup incorrectly.

What if there were no CBV and my system accepted the message and performed a
post smtp recipient validation (such as for our UUCICO gateway people)?
Now the burden is on our system because this site is not reachable to send a
DSN.  Wouldn't it be anti-social that this www1.imc.org has not incurred
overhead on my system?

Luckily, the 452 response allowed the mailserv send mail server to retry and
I was able to see a system that was retrying.

This is a very IMPORTANT concept because in general a real SPAMMER will not
retry to send a rejected message.  This is an undisputable emperical
behavior of spammers.  They might try other new messages, but they will not
try the same message when it was rejected by you.   If this was not the
case, not only the CBV proof of concept is now weaker, but the industry spam
problem would exponentially multiply!

The DMP DNS Lookup for local domain spoof checking or whatever domains the
admin specifies:

DMPTestDomain        winserver.com        ; domains to test with DMP
DMPTestDomain        santronics.com
DMPTestDomain        pan-am.ca

DMP helps eliminates local domain spoofing 100% with my DMP records in my
DNS server.  As I mentioned in other messages, the DNS based lookup proposal
suffer when the domains are unknown and since we know most of the spammers
are unknown, this can be a major overhead on your system.

The RBL lookup does what probably is the #1 anti-spam prevention system
available today using central DNS databases such as bl.spamcop.net.   It
rejects atleat 60% of the total attempts.

And finally the CBV is the final test if it passes the first 3.  There are
my CBV options for it:

;---------------------------------------------------------------------
Section CBV Options              ; Callback Verifier
;---------------------------------------------------------------------

; cbv connect options

NoDomainReject       True        ; Reject if remote has no domain ip (A)
record
NoMXReject           True        ; Reject if remote has no mx record
NoMxTryDomain        True        ; if true, try to connect to domain ip
address

; cbv session options

SocketWaitTime       35                 ; SMTP response wait time
UseNoop              True               ; Send NOOP to help identify WCSAP
UseEhlo              False              ; Try EHLO, fallback to HELO
UseVRFY              False              ; Try VRFY, fallback to RCPT TO
SapHost              mail.winserver.com ; ** USE YOUR MX DOMAIN! **
SapMailFrom          <>                 ; address used for MAIL FROM

; cbv open relay options

UseRSET              True               ; Solves 1 RCPT limit for MAIL FROM:
<>
OpenRelayCheck       True
RejectOpenRelay      True
OpenRelayAddress     [email-masked]

;---------------------------------------------------------------------
Section SMTP Response Codes
;---------------------------------------------------------------------

rescode_filter       554         ; rejected by wcSAP Filter
rescode_dmp          554         ; rejected by DMP Lookup
rescode_rbl          554         ; rejected by RBL lookup
rescode_nodomain     452         ; no domain ip or dns failed
rescode_nomx         552         ; no MX records
rescode_connect      452         ; can not connect
rescode_welcome      452         ; no initial welcome
rescode_helo         552         ; helo/ehlo failed
rescode_from         552         ; MAIL FROM: failed
rescode_rcpt         552         ; RCPT TO: failed
rescode_relay        552         ; Site is an open relay

The result of all this?

For my site,  well, zero spams in the last 12000 transaction attempts the
last 3 days.  These results can not be ignored by me, yet, I am continuing
to see how to optimize the process.   Like you, the idea of a SMTP based
call back leaves a quesy feeling in your stomach.  But damn it, it works!
:-)

Will it work in a large scale deployment?

Of course,  I always see major issues with everyone calling each other all
over the place.  But that does not mean the proof of concept is bad.  It
means that we need to figure out a way to make "return path validation" work
for deployment.  I see a few solutions and its largely based on building
server relationships and/or validation networks I called VISP (Validation
Internet Service Provider).

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 26 Jan 2004 23:12:14 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>
cc: [email-masked]
Subject: Re: "Subject" in the envelope (was: Re: Comments requested on draft-malamud-no-soliciting-04.txt)
Sender: [email-masked]

--On Monday, 26 January, 2004 21:10 -0500 Hector Santos 
<[email-masked]> wrote:

> Thanks for the feedback John.
>
> I agree 100% with each of your points,  most notably having a
> "SUBJ" precede the other commands to pre-empt further
> unnecessary overhead.     About internationalization.
> Admittingly,  I have to study this more.  We are getting hit
> hard from some international customers about this in our Web
> Mail client and just got a reminder the other day.

You might find it interesting to lurk on the "[email-masked]" 
list for a while.  Also you might find it worthwhile to read 
some of the introductory and analysis material in 
draft-klensin-emailaddr-i18n-02.txt (should be posted tomorrow 
or Wednesday -- I'm proofreading now and it is considerably 
different from -01).  There is an attempt at a comprehensive 
analysis of the problem and tradeoffs before the proposal --and 
the analysis may be the more important of the two.

>From my perspective, the internationalization issues are more 
important, long-term, than the spam ones.   Internationalization 
will be with us forever, or at least as long as people use 
different languages and scripts.  I think that by, say, a decade 
from now, we either will have the spam situation under control 
or it will have killed email as we know it --regardless of 
protocols-- as a useful tool.  I am not the only person who has 
reached that conclusion although I have no idea whether we 
constitute any sort of majority or consensus.  You may disagree, 
but that conclusion is the main reason why changes that are 
helpful to both spam-fighting and internationalization tend to 
get more support than one that attacks only one of them 
(especially only spam).

> The only comment I have is that we might lost tracing and
> auditing if it precedes the commands.   If implemented,  an
> system admin may wish to log the rejected transactions by
> atleast FROM and SUBJ, and quite maybe RCPT.

I've often found it helpful to separate the question of "what 
information do we to need to make decisions" from "what do we 
want to log".  I clearly haven't worked through a complete 
analysis, but, for this situation and your proposal, my quick 
reaction is that I'd want to have SUBJ/  MAIL/ RCPT / Trace / 
DATA because:

	* Rejecting on a subject line is technically trivial,
	since it would be lexical and would not require going
	out to the network to look things up.
	
	* MAIL must still precede RCPT, since many folks decide
	on how to respond to RCPT based on the supposed sender,
	and may respond differently to different RCPT commands.
	
	* Any analysis of trace information, history-path
	checking, etc., want to be done, if feasible, before
	DATA is accepted.   Those analysis are often, relatively
	speaking, costly, but may still be cheaper than
	accepting message content.

But, for logging purposes, one could sensibly make the decision 
at SUBJ but accept the MAIL and RCPT commands, not bothering to 
perform any validation, or any non-lexical validation, on the 
latter two if you had already decided to reject the message.

> We learned from our CBV work that it works best to delay the
> MAIL FROM validation until the RCPT is provided.   If RCPT is
> going to refused,  there is no need to perform the CBV.  This
> was an overwhelming major improvement since atleast 60-70%
> recipients attempts on our site are rejected.   In any case, I
> agree. SUBJ should probably precede the other commands and
> implementations can chose when to validate it.

You presumably then do the CBV on the first otherwise-successful 
RCPT command and not before or after that, correct?  That could 
lead to some rather strange patterns of error replies, depending 
on the sequence of RCPT commands, but it might well be worth it.

Out of curiosity, how do you handle the following problem:

Assume that mail is sent to someuser@host1 with a reverse-path 
of maybebogus@odd-domain, and that the relevant DNS records are

   host1 IN MX 10 host2
         IN MX 20 host3

host2 is configured for CBV checking; host3, for whatever 
reason, is not.  The message reaches host3 (it is a bit bigger, 
more responsive, and better connected -- for purposes of 
simplicity, assume there are no firewalls or other middleboxes 
involved).  Host3 tries to pass it along to host2.  When host2 
attempts CBV validation, it gets a DNS timeout on the host1 
domain -- note a timeout, not an authoritative "no" answer. 
Since it is configured to not accept messages unless it can do 
the validation, it returns a 4yz code to host3, resulting in 
host3 requeuing and retrying the message, perhaps for a long 
time.   Host3 considers that behavior mildly antisocial, since 
this is not its problem.

Your thoughts?

> Incidentally, this delayed MAIL FROM validation concept is
> suggested in '2821'.  The same can written about SUBJ.

Yes, I know.  I wrote it :-)

best,
   john

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Mon, 26 Jan 2004 21:35:42 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Martin,

We have added DMP to our anti-spam package and it works great in this area
(checking your local domain and machine spoofs).  WCSAP performs 4 checks:
Internal White/Black List,   RBL,  DMP, and CBV and it is called at the RCPT
state (only if the recipient is acceptable, if refused, wcsap is not
called).

What I did not get 100% clear about SPF, is whether it works from a central
authority (database) concept.  Its documentation seems to indicate both (a
central and your own).  The author should clarify this more.  I am somewhat
resistance to having a "central" database system, however I am not naive to
know that this is probably ultimately the direction the email world will
head to.   So what we did was add a general DNS lookup rules/parser into our
system so that it can ready for anything in the future.

In any case, in my research and testing of the various DNS lookup proposals,
the only "current" benefit I see from all the anti-spam DNS based lookup
proposals is checking spoofs against your own domains.   If you begin to
check other domains, the DNS overhead skyrockets.   It was pointed out to me
by the DMP author that this is a function of your DNS setup. I am not too
sure about this. But I don't pretend to be an DNS expect.  All I know is
that looking up an unknown domain (which is what a majority of the spammers
are) yields a long initial lookup delay.

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Hector Santos" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Monday, January 26, 2004 10:52 AM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

> At 01:36 04/01/26 -0500, Hector Santos wrote:
>
> >Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain
Keys)
> >proposal.  According to recent cyber news,  Yahoo has said they are going
to
> >modify open source mail servers (I believe they mentioned sendmail and
> >qmail) to implement their YDK  and according them, plan to release it
when
> >it all send and done.   I am seriously interested what others think about
> >this.
>
> I have just looked at an article about YDK for a couple minutes, and
> don't claim to understand it fully. There seem to be some similarities
> to SPF and other proposals. We (W3C) have deployed SPF records just
> recently. The main benefit we are expecting is that we can avoid
> ourselves and our mailing lists being spammed by impersonators faking
> our own email addresses. That's not all of spam, but it's a very
> nasty and troubling bit of it.
>
> Regards,   Martin.
>

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: "Subject" in the envelope (was: Re: Comments requested on draft-malamud-no-soliciting-04.txt)
Date: Mon, 26 Jan 2004 21:16:09 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Thanks for the feedback John.

I agree 100% with each of your points,  most notably having a "SUBJ" precede
the other commands to pre-empt further unnecessary overhead.     About
internationalization.  Admittingly,  I have to study this more.  We are
getting hit hard from some international customers about this in our Web
Mail client and just got a reminder the other day.

The only comment I have is that we might lost tracing and auditing if it
precedes the commands.   If implemented,  an system admin may wish to log
the rejected transactions by atleast FROM and SUBJ, and quite maybe RCPT.

We learned from our CBV work that it works best to delay the MAIL FROM
validation until the RCPT is provided.   If RCPT is going to refused,  there
is no need to perform the CBV.  This was an overwhelming major improvement
since atleast 60-70% recipients attempts on our site are rejected.   In any
case, I agree. SUBJ should probably precede the other commands and
implementations can chose when to validate it.

Incidently, this delayed MAIL FROM validation concept is suggested in
'2821'.  The same can written about SUBJ.

Thanks again.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Hector Santos" <[email-masked]>;
<[email-masked]>; <[email-masked]>
Sent: Monday, January 26, 2004 7:17 PM
Subject: "Subject" in the envelope (was: Re: Comments requested on
draft-malamud-no-soliciting-04.txt)

> Hector,
>
> I just wanted to comment on one aspect of your proposal
> outline...
>
> --On Sunday, 25 January, 2004 06:40 -0500 Hector Santos
> <[email-masked]> wrote:
>
> > However,  it
> > is too restrictive to a particular need where the same result
> > can be achieved with a simple SUBJECT line.    The SUBJECT
> > line idea can address this and many other neat concepts such
> > as overhead reducing mail driven events. For example,
> > subcribing to a mail list:
> >
> >         EHLO mailer.foobar.com
> >         MAIL FROM: <[email-masked]>
> >         RCPT TO:  [email-masked]
> >         SUBJECT: subscribe
> >         QUIT
> >
> > The feature and product benefits  offered by a SUBJECT line
> > design change is much greater and wider than this specific
> > EHLO/MAIL FROM design change which targets only one concept.
>
> If one really wanted to do this, and I'm not yet convinced,...
>
> (1) You need to work out a number of details, preferably in the
> form of an Internet-Draft, if you expect this to go anywhere.
> For example, Subject lines can get pretty long, and there is no
> equivalent to 822 continuation lines in SMTP sender dialogues.
> Some of the internationalization issues also raise coding
> questions, and it isn't clear to me that the right answers for
> 2822 would be the right answers for the envelope.
>
> (2) You will find that it is easier to establish
> interoperability with a wider range of SMTP servers if you
> confine the verb to four characters, e.g., SUBJ and not SUBJECT
>
> (3) I would think that you would want SUBJECT to precede RCPT
> and possibly MAIL, not follow them.  If you are trying to
> conserve resources, it would be good to not need to validate
> RCPT commands (or give the sender a clue about what is and is
> not valid), and probably not even apply whatever testing is done
> to the return path, if you are going to reject the transmission
> on the content of a subject command.
>
>      john

Date: Mon, 26 Jan 2004 19:17:49 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked], [email-masked]
Subject: "Subject" in the envelope (was: Re: Comments requested on draft-malamud-no-soliciting-04.txt)
Sender: [email-masked]

Hector,

I just wanted to comment on one aspect of your proposal 
outline...

--On Sunday, 25 January, 2004 06:40 -0500 Hector Santos 
<[email-masked]> wrote:

> However,  it
> is too restrictive to a particular need where the same result
> can be achieved with a simple SUBJECT line.    The SUBJECT
> line idea can address this and many other neat concepts such
> as overhead reducing mail driven events. For example,
> subcribing to a mail list:
>
>         EHLO mailer.foobar.com
>         MAIL FROM: <[email-masked]>
>         RCPT TO:  [email-masked]
>         SUBJECT: subscribe
>         QUIT
>
> The feature and product benefits  offered by a SUBJECT line
> design change is much greater and wider than this specific
> EHLO/MAIL FROM design change which targets only one concept.

If one really wanted to do this, and I'm not yet convinced,...

(1) You need to work out a number of details, preferably in the 
form of an Internet-Draft, if you expect this to go anywhere. 
For example, Subject lines can get pretty long, and there is no 
equivalent to 822 continuation lines in SMTP sender dialogues. 
Some of the internationalization issues also raise coding 
questions, and it isn't clear to me that the right answers for 
2822 would be the right answers for the envelope.

(2) You will find that it is easier to establish 
interoperability with a wider range of SMTP servers if you 
confine the verb to four characters, e.g., SUBJ and not SUBJECT

(3) I would think that you would want SUBJECT to precede RCPT 
and possibly MAIL, not follow them.  If you are trying to 
conserve resources, it would be good to not need to validate 
RCPT commands (or give the sender a clue about what is and is 
not valid), and probably not even apply whatever testing is done 
to the return path, if you are going to reject the transmission 
on the content of a subject command.

     john

Date: Mon, 26 Jan 2004 15:12:20 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'A No Soliciting SMTP Service Extension' to Proposed Standard
Sender: [email-masked]

multiple layers of response here:

Layer 1:  generally about standardizing spam countermeasures

Part of the difficulty with standardizing spam countermeasures is that
there's probably no comprehensive solution to the spam problem - thus
any proposal that was not grandiose in scope would arguably fail the "no
known technical omissions" test.  Unfortunately, there is a potential
for partial solutions to conflict with one another - either because
they're trying to solve the same problem in different ways leading to a
lack of uniformity, or because use of one spam countermeasure precludes
another one.

A lot of the spam countermeasures in use seem to me to be actually 
harmful in that they decrease the ability of the mail system to deliver
legitimate messages reliably.  Even if you consider that not filtering
spam also affects mail reliability (either because it overloads mail
systems or because it overloads recipients who might accidentally 
delete a legitimate message embedded in a large batch of spam), some
spam countermeasures still seem to do more harm than good.  User
experience varies - what works well for one group of users may not work
well for another.

If we're going to start standardizing spam countermeasures I'd like to
see IESG or the Apps area adopt a strategy for dealing with the variety
of proposals I suspect we're going to see, as various groups seek to 
gain legitimacy for their work.  For instance:

1.  Since I doubt we'll be able to decide exactly which set of 
countermeasures should be widely implemented, my guess is that we'll
end up publishing several competing proposals at Experimental and/or
Proposed and re-evaluating them after a time in light of operational 
experience and market acceptance.   If we are going to do that we should
publicly say so up front, so that we have some semblance of uniformity
and we aren't as likely to be accused of favoritism.

2.  Some spam countermeasures are "riskier" than others, in that they
use less-than-reliable criteria for classifying mail as spam, or they
adversely affect mail reliability for other reasons, or they change
SMTP or the mail format in inappropriate ways.  If we are going to
publish risky proposals I'd strongly prefer that we publish them as
Experimental rather than Proposed.  Proposed standards should "do no
harm".  

However there still might be some value in publishing risky proposals
as Experimental - 

First, to invite review of those proposals by experts within IETF and 
where feasible, to encourage modification of those proposals to minimize
their potential for harm

Second, to encourage modification of those proposals so that conflicts 
between them are minimized; 

Third, to provide the opportunity to clearly label which proposals are
seen to be risky and for what reasons

Fourth, to encourage better collection of experience with
implementations of such proposals  (since people are already widely
using risky countermeasures, we can't expect to stop them - but we might
as well try to learn from it)
----------------------------------------------------------------------
Layer 2: this proposal specifically

1. The basic idea seems good and "mostly harmless", but I think it needs
substantial revision before being adopted.  With appropriate changes,
I'd recommend it for Proposed PROVIDED there's a pre-announced policy
on how we're going to deal with similar proposals.  I don't think we
should be appearing to endorse this specific approach over other
approaches at this time.

2. [section 2] Nit: to me it seems confusing for solicitation-keywords
to be a single ehlo-param (in RFC 1869/2821 parlance) and for the
keywords to be comma-separated.  I'd like to see us keep EHLO responses
fairly simple and uniform in format.  Have we had comma-separated
keywords in other EHLO responses?

I guess the idea is to keep one syntax for "solicitation-keywords"
that can be used in both EHLO response and MAIL FROM command.  That
might be better than having two separate grammars.

3. [section 2] Nit: it would be better if the entire grammar for
the EHLO response were in one place, rather than spread over
multiple sections.

4. [section 2.1] Nit: the last paragraph seems out of place;
I think it should go in an acknowledgements section near the 
end of the document.

5. [section 2.2] I understand the need to define some initial 
solicitation classes but I'd like to see us avoid endorsing 
these.  The MAPS-UBE class in particular, while it makes an
interesting topic for discussion, doesn't strike me as something
that we should endorse by making it a built-in part of a standards-
track protocol.  2.2.1 does say "for illustrative purposes only"
but similar disclaimers in other standards have been disregarded.

More generally, what I'd like to see is a separation between the
specification of the protocol engine (i.e. the grammar) and the
specification of solicitation classes.   (I realize that we've
traditionally included some keywords in grammars for other protocol
specifications.  I think we need to stop doing this, as it makes
the grammars more complex and less useful for both protocol
implementation and constructing protocol verifiers.)

So the protocol engine is what should be standardized, and we should
have some initial solicitation classes registered that are not part of
the standard (they could go in an appendix to the document labelled
"initial solicitation class definitions" that says "not part of the
standard").

It would be okay with me to use some of the initially defined classes
in examples.

Finally I think this is one of those cases where x- fields might not
be a good idea - at least, I'd like to encourage public review of
new solicitation classes.

The grammar should be rewritten to reflect these various changes.  For
instance:

     Solicitation-keywords = registered-word 0*("," registered-word)

     registered-word = ALPHA 0*(wordchar)
                                  ; registered-word(s) are registered
                                  ; with the IANA
     wordchar = ("-" / "_" / ":" / ALPHA / DIGIT)

6. [section 2.3]  Is it possible that the recipient doesn't want to
disclose the reasons for rejecting the mail?  Perhaps there should
be some generic indication that the message was rejected due to 
policy rather than because the recipient address was not valid.
(550 is the right code but is probably overused.)  Or maybe this 
proposal should just require ENHANCEDSTATUSCODES as 5.7.1 is a 
clear indication that this is a policy violation without saying 
exactly what policy is violated.

If nothing else this should make clear whether the policy indication
is in some machine-readable format (is it part of the protocol?
if so, what's the precise syntax?)  or is meant to be human readable.

Is it conceivable that a recipient might be willing to accept some kinds
of solicitation but impose various limitations or quotas on that kind
of mail?  In this case a 4xx error code and 4.7.1 status code might be
appropriate to indicate "go away, come back later".

7. [section 2.6] I have strong objections to various parts of this
section:

- Most of this proposal is about giving the sender a way to assert
that a message is a solicitation, and that seems to me to be 
"mostly harmless".  Giving intermediaries a way to make similar
assertions is "risky" and I don't think such a practice should be
standardized. 

For example: a sender can assert (rightly or wrongly) that an
advertisement is solicited by a recipient and be in a position to know
whether that assertion is true; an intermediary cannot be expected to
know whether such an assertion is true.

- Letting intermediaries' assertions get confused with the sender's
assertion (as provided for in section 2.x) is even worse, and I'd regard
that as something that should not even be published as Experimental.

- Even accepting that it's useful for intermediaries to make assertions
about the solicitation classes of a message, it's not at all clear
that the classes that can be asserted by an intermediary are similar
to those that can be asserted by a sender.  An intermediary can
make assertions about content (e.g. "this message contains a GIF image")
but is not in a good position to make assertions about intent.  

- Even accepting that it's useful for intermediaries to make assertions
about the solicitation classes of a message, I don't think the Received 
field is a good place to do that, and I don't like the idea of storing
protocol elements in comments or of having MTAs read and trust the 
contents of comments.   MTAs should as much as possible treat message
content as opaque.  

I strongly recommend that this document NOT provide a way for 
intermediaries to make assertions about solicitation classes,
as that seems orthogonal to the main purpose of this document,
and it seems to make it much riskier.

Even accepting that it is useful for intermediaries to make 
assertions about solicitation classes, the protocol for doing this
should be defined in a separate document, and those assertions should be
kept cleanly separated from assertions made by the message originator,
and there should probably be different classes for those indications.

8. [section 2.7] The wording is muddy.  It appears to apply only
to SYSTEM-WIDE, and that doesn't seem right.

IMHO, anytime a server asserts NO-SOLICITING in EHLO, this means:

- if NO-SOLICITING SYSTEM-WIDE is asserted and a set of prohibited
solicitation classes is included, client SHOULD NOT attempt to relay
mail for which the SOLICIT=  MAIL FROM parameter includes a class that
is prohibited in the EHLO response.

  (however if the client does so it's still the server's responsibility
  to reject it)

- regardless of whether a set of solicitation classes is included
  in the EHLO response, client MUST convey SOLICIT= faithfully (i.e. 
  as received from the previous hop) in MAIL FROM for each message 
  relayed to a server that supports NO-SOLICITING.

9.  Actually I suggest getting rid of the SYSTEM-WIDE and PER-RECIPIENT
keywords.  Here's why:

- it's useful to allow a combination of SYSTEM-WIDE and PER-RECIPIENT
  policies, rather than to force one or the other.  The SYSTEM-WIDE
  policies would apply to all recipients supported by that server,
  but individual recipients could have more restrictive policies.

- it's useful for an MTA to have the ability to relay SOLICIT=
indications even if it doesn't actually impose system-wide or 
per-recipient restrictions.  Yes, it could claim PER-RECIPIENT
and then never impose any restrictions, but that seems misleading.

I'd also suggest changing the name of the EHLO keyword because
supporting relaying of SOLICIT= doesn't imply that the MTA has any
restrictions on soliciting.   So I suggest:

- rename the EHLO keyword something like "RESTRICT"
- allow the EHLO keyword to have system-wide options.  e.g.

   RESTRICT means that the SOLICIT= keyword is supported but that
   no system-wide restrictions are advertised, and

   RESTRICT=class1,class2,... means that the SOLICIT= keyword is
   supported and that class1,class2,... are prohibited on a
   system-wide basis.

10. Somewhere some text like the following seems appropriate:

Note that the system-wide option, while useful, is of marginal
value.  Many MTAs want to be able to accept mail for multiple domains
that may have different policies, and any large user community will
probably need to have different policies.  Typically there are some
people in an enterprise who _need_ to be able to receive advertisements-
e.g. people who do procurement and who naturally need to know what goods
are available at what prices from which sources.

11. [section 2.8]  The ability to relay SOLICIT= SHOULD be enabled by
default. However the default configuration SHOULD NOT impose any 
system-wide or per-recipient policy.

An MTA that advertises the EHLO keyword MUST relay SOLICIT= faithfully
to any next-hop MTA that advertises the extension in EHLO.

An MTA that advertises the EHLO keyword MAY implement the ability to
advertise a system-wide policy in EHLO.  If such a policy is advertised
it MUST be enforced by rejecting messages for which MAIL FROM asserts
a solicitation class that conflicts with that policy.  (It MAY also
be enforced by other mechanisms, e.g. looking at the Solicitation: 
field and rejecting messages for which that field conflicts.)

An MTA MAY also implement a system-wide policy without advertising 
all of that policy in EHLO.

An MTA that advertises the EHLO keyword MAY implement per-recipient
policies for zero, some, or all of its recipients.

12. [section 3]  I think this belongs in an appendix, as it seems
to be non-normative.

13. It is fairly obvious that sites and recipients will want policies
based on reliable characteristics of the message other than a
solicitation class advertised by the sender.  e.g. I want to be able to
reject application/ms-word documents.  And then we also have things like
the content-negotiation stuff that the fax people want.  Given this,
what is the potential for conflicts between these various policy
mechanisms?  And should we try to sort them out now or wait until later
when we have more experience with them?

-- 
Keith Moore                              http://www.cs.utk.edu/~moore/

Date: Mon, 26 Jan 2004 10:52:53 -0500
To: "Hector Santos" <[email-masked]>, <[email-masked]>, <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

At 01:36 04/01/26 -0500, Hector Santos wrote:

>Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain Keys)
>proposal.  According to recent cyber news,  Yahoo has said they are going to
>modify open source mail servers (I believe they mentioned sendmail and
>qmail) to implement their YDK  and according them, plan to release it when
>it all send and done.   I am seriously interested what others think about
>this.

I have just looked at an article about YDK for a couple minutes, and
don't claim to understand it fully. There seem to be some similarities
to SPF and other proposals. We (W3C) have deployed SPF records just
recently. The main benefit we are expecting is that we can avoid
ourselves and our mailing lists being spammed by impersonators faking
our own email addresses. That's not all of spam, but it's a very
nasty and troubling bit of it.

Regards,   Martin.

Date: Mon, 26 Jan 2004 07:13:01 -0800 (PST)
From: "[email-masked]" <[email-masked]>
To: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

On Mon, 26 Jan 2004, Nathaniel Borenstein wrote:

> Another comment:  I strongly urge moving this discussion to the new 
> mail-ng mailing list once it's up and running.  The imaa and smtp 
> mailing lists will make a lot more short-term progress if the 
> longer-term discussions have their own venue.  -- Nathaniel

I've tried forwarding original draft and couple recent comments which are 
directly on topic of new mail system to this new mail list in hopes of 
moving those discussions there, but system bounced it off with automated 
autoreply. Hopefully list owner can do it himself if he things its worth 
it to start off that list with. Here are the posts I wanted to forward:
 http://www.imc.org/ietf-smtp/mail-archive/msg00927.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00928.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00929.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00935.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00943.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00945.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00950.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00951.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00952.html
 http://www.imc.org/ietf-smtp/mail-archive/msg00954.html

-- 
William Leibzon
Elan Networks
[email-masked]

Date: Mon, 26 Jan 2004 08:54:17 -0500
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Cc: Keith Moore <[email-masked]>, John C Klensin <[email-masked]>, <[email-masked]>, <[email-masked]>
To: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

On Sunday, January 25, 2004, at 08:47  PM, [email-masked] wrote:

> And for designers, programmers, and quicker implementation its quite a 
> bit
> better if new protocol and old one share certain components. For 
> example
> MIME encoding/decoding, etc.

Well, let's not rule out improvements to MIME in the process.  The name 
MIME includes a lot of things, from the two-level content-type 
architecture to the quoted-printable and base64 transfer encodings.  
I'd certainly like to see us keep the former, but the latter can only 
be seen as warts for backward compatibility with SMTP/822.  I wouldn't 
make it one of the major goals of email NG, but I certainly wouldn't 
mind seeing the new protocol eliminate the need for 
content-transfer-encodings.

Another comment:  I strongly urge moving this discussion to the new 
mail-ng mailing list once it's up and running.  The imaa and smtp 
mailing lists will make a lot more short-term progress if the 
longer-term discussions have their own venue.  -- Nathaniel

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Mon, 26 Jan 2004 02:55:44 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I agree with you. I believe SMTP can be used to offer a "dual" state
transaction model.   Yes, design and implementation confusion can be an
issue. However, I believe this only means it would become one of many
aspects for the new design to address and minimize.

I would like to suggest an approach borrowing old concepts such as Fidonet
when it was faced with new industry needs and state machine protocols.

The original and standard protocol was called FTSC1.   This was mostly based
on a kludged XMODEM handshake. A second one called WAZOO offered an easier
session handshake and introduced ZMODEM for data transfer.  A 3rd one called
EMSI addressed the session handshake issues. EMSI introduced text based tags
or attributes that defined and establish the compatibility level between the
client and server.  It also introduced new tags to define new file transfer
protocols.

If EMSI or WAZOO was not established,  the fallback was FTSC1.    As a side
note, for historical perspective, the demise of Fidonet was not helped by
the fact that the Fidonet or FTSC (Fidonet Technical Standards Committee)
protocol police insisted on backward compatibility.  WAZOO/EMSI flexibility
allowed for growth including new "internet readiness."  As new developers
come aboard to support the new direction (internet), most used the easier to
implement WAZOO/EMSI protocols and avoided FTSC1 mostly due to the fact it
was a complex outdated modified XMODEM7 binary protocol handle shake which
is probably akin to the idea of SMTP having no states other than DATA!   It
didn't fit well in packet switching networks. Security was weak and it
created conflicts between the development community and the FTSC.  Fidonet
operators or members found to be using non-FTSC1 compliant mailers were
excommunicated from Fidonet.  To get an Fidonet address (akin to getting a
MX record maybe),  your HOST (akin to maybe an ISP) would check for FTSC1
compliance.  Of course, IETF does not have strong "network" membership
requirements such as this, but I would hate to see the demise of the IETF
because it could not or was slow to address the current industry issues
and/or needs. It is all deja vu when you hear news that YAHOO is taking the
lead in changing the SMTP specification.

In any case,  WAZOO/EMSI was established with the SERVER answering with
special greeting allowing for EMSI or WAZOO detection.  The compatible
client detected either one and choose its method.  If the server did not see
a response within X seconds, it finally issued a FTSC1 detection packet.

With SMTP, I believe it would be a lot easier because SMTP is already
Text/Tag based.

The initial greeting 220 line can be used to define the SMTP attributes.  In
a way, we sort of having something like this now.   The BCP for the greeting
line is to use a "ESMTP" tag to indicate an Extended SMTP ready server.
However, I am not sure if anyone uses this since '2821' requires you to use
EHLO first anyway.

But we can use the greeting to define the server attributes which compliant
clients can read and use.

In Fidonet, to support WAZOO, WAZOO tag was used in the server greeting.
For EMSI, it used a EMSI tag. In addition, if EMSI was detected a handshake
was done to exchange and establish the session attributes.

With SMTP, it is easier:

examples:

220 host,  SMTP, SMTP3 [product info] Server Reader
220 host,  SMTP3 [product info] Server Reader

SMTP indicates old SMTP state support.   SMTP3 indicates the new state
support.    Non-compliant clients would not know the difference.  Only
clients looking for SMTP3 will know the level of support this server offers.
Whether backward support is required, I leave it to you guys to debate.
Please just remember what I noted above.  Confusion and conflicts is reduced
when developers (old or new) do what is only demanded by the market needs.
If old support is mandatory and is conflictive with security issues, then it
needs to be discussed whether old support is going to complicate migration
and acceptability.  The concept of allowing it to be option for system
administrators to decide should be discussed.

Once the SMTP3 session is established, the new state machine prevails!

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
Sent: Sunday, January 25, 2004 7:27 PM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

>
> I'm all for separating the envelope from the message content, but I'd
> be interested in investigating an even more radical change - (probably)
> scrapping SMTP entirely or (less likely) branching out of the SMTP
> state machine very early.  Or in other words, I could see making the
> new protocol similar enough to SMTP that a single server could
> implement both, but that might just invite confusion.
>
>

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Mon, 26 Jan 2004 01:36:00 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Don't wish to insult anyone with the obvious but here some points we should
keep in mind when considering revamping SMTP.

1) "worthy new solutions" are those that best address real problems.  While
there will also be a market where needless change is performed ("latest toy
syndrome"),  in general, costly revamps become more feasible for
consideration when they provide major benefits.

2) Consideration is enhanced when new functional specs include well outlined
migration, acceptance and deployment plans, ideally including or referencing
source code examples and/or API, SDK tools to help the migration process.

3) Responsive vendors will help the process by making the "conversion"
process easier via their software, and lastly

4) Where there is a market for change, a tertiary market of tools (proxies,
middle ware, APIs) will be made available to address legacy software and/or
provide easier development and migration.

In short, "lets not fear change"

Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain Keys)
proposal.  According to recent cyber news,  Yahoo has said they are going to
modify open source mail servers (I believe they mentioned sendmail and
qmail) to implement their YDK  and according them, plan to release it when
it all send and done.   I am seriously interested what others think about
this.

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 26 Jan 2004 04:57:35 +0100
To: [email-masked], Keith Moore <[email-masked]>
From: "J-F C. (Jefsey)  Morfin" <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Cc: John C Klensin <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I an idea we work on is to keep SMTP as it is, and use it for mail 
signaling (weemail). The interest is that nothing is to be changed except 
to modify slightly the user agent and develop a new agent - a store and 
retrieve mail server which may also be used as a gateway between the 
current and the new system. The "you've got mail" is then not refering to a 
piece of junk with emebeded virus you have been forfced to accept into your 
computer and spend bandwidth for, but to a file of any kind  you can chose 
to disregard and leave on the sender's server or read totally or in part, etc.

This does not kill spam as there are ways to fake a server. But it makes 
wild spam far more complex and drastically reduces the bandwidth usage.

The possibilties it easily allows, like dynamic distribution lists, also 
permits to prioritize the reader interest and to leave the spam at the 
bottom of the basket. Mail URLs also allows to more easily hunt for known 
spam or to disregard known commercial sources - or already read posts.

It is there quite easy to use 0-Z numbering to name LHS and to have a 
resolution system into any vernacular (ML, menus, emoticon, etc.).
jfc

At 02:47 26/01/04, [email-masked] wrote:

>First of all end-users would not care much what protocol it is when
>"you've got mail" pops up.
>
>And for designers, programmers, and quicker implementation its quite a bit
>better if new protocol and old one share certain components. For example
>MIME encoding/decoding, etc. It would faciliate process  if protocol can
>be implemented on same mail server software that would be able to decide
>based on certain dns or other parameters what protocol to use when sending
>email to the other end.
>
>And look at the example of IPv4 vs IPv6 - IPv6. While there are substantial
>improvements in features available with IPv6, as far as implementation and
>design and application programmings, its not that different.
>
>On Sun, 25 Jan 2004, Keith Moore wrote:
>
> >
> > I'm all for separating the envelope from the message content, but I'd
> > be interested in investigating an even more radical change - (probably)
> > scrapping SMTP entirely or (less likely) branching out of the SMTP
> > state machine very early.  Or in other words, I could see making the
> > new protocol similar enough to SMTP that a single server could
> > implement both, but that might just invite confusion.

Cc: Keith Moore <[email-masked]>, John C Klensin <[email-masked]>, <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Sun, 25 Jan 2004 20:00:18 -0500
To: [email-masked]
Sender: [email-masked]

> First of all end-users would not care much what protocol it is when
> "you've got mail" pops up.

No, but they might care about having email work more reliably.

> And for designers, programmers, and quicker implementation its quite a 
> bit
> better if new protocol and old one share certain components. For 
> example
> MIME encoding/decoding, etc. It would faciliate process  if protocol 
> can
> be implemented on same mail server software that would be able to 
> decide
> based on certain dns or other parameters what protocol to use when 
> sending
> email to the other end.

Yes that's why I would consider running both protocols on the same port.

On the other hand if the new protocol were designed well then there 
would be less need to share components.  For instance if all mail were 
binary transparent and delivered directly from the sender's submission 
server to the recipient's message store (which is close to what you 
need for reliable error reporting) there would be not be as much need 
for such an MTA to encode or decode MIME.

> And look at the example of IPv4 vs IPv6 - IPv6. While there are 
> substantial
> improvements in features available with IPv6, as far as implementation 
> and
> design and application programmings, its not that different.

Not at first glance.  But for a variety of reasons (e.g. address 
scoping and new kinds of hosts) I suspect we'll end up programming 
differently in IPv6 than in IPv4.

Date: Sun, 25 Jan 2004 17:47:25 -0800 (PST)
From: [email-masked]
To: Keith Moore <[email-masked]>
cc: John C Klensin <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

First of all end-users would not care much what protocol it is when
"you've got mail" pops up.

And for designers, programmers, and quicker implementation its quite a bit 
better if new protocol and old one share certain components. For example 
MIME encoding/decoding, etc. It would faciliate process  if protocol can 
be implemented on same mail server software that would be able to decide 
based on certain dns or other parameters what protocol to use when sending 
email to the other end.

And look at the example of IPv4 vs IPv6 - IPv6. While there are substantial 
improvements in features available with IPv6, as far as implementation and 
design and application programmings, its not that different. 

On Sun, 25 Jan 2004, Keith Moore wrote:

> 
> I'm all for separating the envelope from the message content, but I'd 
> be interested in investigating an even more radical change - (probably) 
> scrapping SMTP entirely or (less likely) branching out of the SMTP 
> state machine very early.  Or in other words, I could see making the 
> new protocol similar enough to SMTP that a single server could 
> implement both, but that might just invite confusion.

Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Sun, 25 Jan 2004 19:27:35 -0500
To: John C Klensin <[email-masked]>
Sender: [email-masked]

I'm all for separating the envelope from the message content, but I'd 
be interested in investigating an even more radical change - (probably) 
scrapping SMTP entirely or (less likely) branching out of the SMTP 
state machine very early.  Or in other words, I could see making the 
new protocol similar enough to SMTP that a single server could 
implement both, but that might just invite confusion.

From: Carl Malamud <[email-masked]>
Subject: Re: Comments requested on draft-malamud-no-soliciting-04.txt
To: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 25 Jan 2004 08:02:36 -0800 (PST)
CC: [email-masked], [email-masked], [email-masked], [email-masked]
Organization: Memory Palace Press
Sender: [email-masked]

> 
> [email-masked] writes:
> > Let the discussion begin...
> 
> It strikes me that many senders will want to pick at most one category, 
> which requires receivers to add the entire IANA registry in the SMTP 
> server configuration in order to avoid spam. That's not fortunate.
> 

Hi Arnt -

Good comment.  I hope that's not a huge issue ... one purpose of a
designated expert is to try and reduce the number of registrations
to something manageable.  Note also the trailing wildcard provision
as a second mechanism to reduce the registry size.

In term of the EHLO command, you are correct you need to load and
dump if you want a blanket "I don't want anything."  In practice,
I would think most implementations would allow users to load
"what matters" (e.g., the most common half-dozen), then perhaps
log unknown keywords as they come across the wire, allowing users
to add new classes as they become common.

I think the classes you'll see in a Solicitation: header will be
very few ... it takes forever to pass a law.  You'll find more
in the received: headers, but you'll probably only care about
the ones your particular filtering software uses.

If people think this is an issue, there's a couple of things we
could do.  The easiest would be a machine-readable (e.g., xml)
registry that could be retrieved at install, run-time, or when
the user reads mail.  I purposely didn't spec that in as that
isn't how the IANA currently maintains registries and, rather
than push that particular envelope, it seemed easier to allow
developers to decide how they want to handle that in practice.

Regards,

Carl

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Subject: Re: Comments requested on draft-malamud-no-soliciting-04.txt
Date: Sun, 25 Jan 2004 06:40:43 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

This falls under anti-spam which IMO currently represents the main impetus
for SMTP "change."

There are many anti-spam proposals that imposes change.  Because of this,
the technical design approach we used was to analyze all proposals
(including Malamud's proposal) and see how they are all related to each
other and also addressed other design goals. The end was result was a new
generalized SMTP model for our mail server providing two new design
features:

1) 3rd party programmable logic to process new commands with the expectation
of new possible SMTP commands introduced (including our own), and

2) A "Black Box" access control hook system at each state from connection to
quit.

In addition,  the CAN-SPAM Act is seriously viewed as a very important item
that we must support.   In my technical review of the CAN-SPAM Act,  there
are two basic technical mandates that relates to SMTP:

    1) Verifiable Return Addresses
    2)  Subject/Topic Identification

Although there is a 3rd mandate (opt-out capability for the user), I don't
believe this is applicable to SMTP unless some standard opt-out trigger/tag
automation concept is introduced.

A summary of the CAN-SPAM Act can be viewed at:
http://www.spamlaws.com/federal/summ108.html#s877

Item #1 (Return Paths) is already addressed in our new design.  Our "Black
Box" hook allows us to provide validate the data pass to it.  In this case,
the return path along with other already established data (connection IP and
machine domain, etc).   Currently, we use a CBV concept only because its the
only thing currently possible and proven to work with the current SMTP
protocol.   I don't wish to rehash the issues related to it but to simply
note it is the CONCEPT that is important.  It is a BLACK BOX and thus it
allows a future replacement when a more ideal IETF endorsed method is
invented.

Item #2 (Topic Identification) is the main issue right now that is not
possible with the current specs and for this reason, I believe it is one
aspect about the CAN-SPAM Act that has the spammers licking their chops.  As
you can read in the summary referenced above, a conflictive statement is
made with the parenthetical clarification of the labeling requirement:

        The Controlling the Assault of Non-Solicited Pornography and
Marketing Act requires
        unsolicited commercial e-mail messages to be labeled (though not by
a standard method) .....

I believe this is one item the IETF should address when refinement input is
provided to the FCC.  We need to remove this potential "non-standard
labeling" loop hole.  I will address this again below.

Currently, the only way possible for a new SMTP system to offer a system
policy to address Subject/Topic Identification processing is to allow the
data to be transmitted.  Spammers provide a higher weight  to email
addresses when DATA can be delivered (transaction completed).  This is based
on a spammer's statistics report that was leaked out by 'some dumbass spam
company' [sic] to an ASRG participate showing their entire year-to-date spam
history. Interested parties can check out the sapmmer data here:
http://www1.ietf.org/mail-archive/working-groups/asrg/current/msg08868.html

In our particular case, our "hook" into the DATA state will provide us with
a way to offer operators or 3rd party developers to perform Subject/Topic
Identification filtering.

Also, there seems to be a "disturbing trend" currently perpetuated by YAHOO
to permits spammers to complete their transactions and to perform all
checking at the DATA state.  For YAHOO, this DATA state validation
processing currently includes performing the final local user recipient
validation since it has allows for all RCPT addresses to be initially
accepted at the RCPT state. Since YAHOO is also promoting their Domain Key
proposal which a functional specification we can currently classify as
"vapor-specs," they have a high interest instilling the idea that all
validation should be performed at the DATA stage.

Now, how all this relate to Malamud's proposal?

Well, when I first reviewed the proposal (rev 02) I wanted to see if the
proposal addressed the CAN-SPAM Act  topic identification mandate.

According to the proposal's current class keywords, there are 3
classifications:

   MAPS-UBE       Unsolicited Bulk Email
   ADV                   Unsolicited Commercial Email
   ADV:ADLT        Sexually Explicit Commercial Mail

Per the proposal, the classification is provided as MAIL FROM: extended
attributes.  So it seems that this proposal will help SMTP developers
provide system policy "CAN-SPAM" Ready features in thier software.

Of course, the obviously problem as with all new SMTP altering proposals,
this will only apply to compliant clients.  However,  what is important to
realize is due to the non-standard labeling loophole in the Act,  spammers
will have no incentive nor legally required reason to comply with this
method.

This is reason why I have repeatedly stated that we need to seriously review
the SMTP functional specification and provide "enforcement" concepts.
Unless the fundamental philosophy and "relaxed spirit" of SMTP functional
specification is reviewed to begin offer developers the ability to enforce
compliance,  we seriously minimize the acceptance and deployment of any
anti-spam proposal.

Of course, we still have the ability to do topic identification at the DATA
state so all is not lost by any means, but the impact is this proposal (and
all others) is reduced drastically.   In short,  designers will need to do
provide two logics to address topic identification.

Fortunately, there is one item in CAN-SPAM Act that allows us to provide us
a direction of compliancy enforcement.  CAN-SPAM section 11.2 says:

SEC. 11. IMPROVING ENFORCEMENT BY PROVIDING REWARDS FOR INFORMATION ABOUT
VIOLATIONS; LABELING

        (2) a report, within 18 months after the date of enactment of this
Act,
        that sets forth a plan for requiring commercial electronic mail to
be
        identifiable from its subject line, by means of compliance with
Internet
        Engineering Task Force Standards, the use of the characters `ADV' in
the
        subject line, or other comparable identifier, or an explanation of
any
        concerns the Commission has that cause the Commission to recommend
        against the plan.

So there you have it.  We have the opportunity to do this correctly.

In summary, I throwing this out for thought:

Do we tell the FCC that spammers must comply with "SMTP 821.3" which
includes support for Malamud's "A No Soliciting SMTP Service Extension"
with allowance for 6 month (or 1 year)  time frame for compliance?
Basically my concern here is that unless the IETF addresses compliancy
enforcement,  SPAMMERS can use this to legally side step support for any new
SMTP extended protocol.

Is the proposal too complicated creating an acceptance and deployment
barrier?  and if so,  do we help spammers "simplified" their efforts by
offering a new simpler SUBJECT command concept that can be issued before
(RCPT or DATA)?

I personally like the SUBJECT idea because it keep the concept open and
flexible for other automation ideas.  Malamud's proposal is well written and
I believes helps address the main issue it attempts to address. However,  it
is too restrictive to a particular need where the same result can be
achieved with a simple SUBJECT line.    The SUBJECT line idea can address
this and many other neat concepts such as overhead reducing mail driven
events. For example, subcribing to a mail list:

        EHLO mailer.foobar.com
        MAIL FROM: <[email-masked]>
        RCPT TO:  [email-masked]
        SUBJECT: subscribe
        QUIT

The feature and product benefits  offered by a SUBJECT line design change is
much greater and wider than this specific EHLO/MAIL FROM design change which
targets only one concept.

I will support Malamud's proposal if endorsed,  but I like something like a
SUBJECT command from a design standpoint and greater benefits it offers.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: <[email-masked]>
To: <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Saturday, January 24, 2004 6:14 PM
Subject: Comments requested on draft-malamud-no-soliciting-04.txt

>
> Folks,
>
> A couple of weeks ago I received a request from the author to progress
> draft-malamud-no-soliciting-04.txt to proposed standard. I reviewed the
> document and sent a rather long list of corrections and suggestions to the
> author, most of which have been incorporated into the -04 version.
>
> Here's the document abstract:
>
>    This Internet-Draft proposes an extension to SMTP for an electronic
>    mail equivalent to the real-world "No Soliciting" sign. In addition
>    to the service extension, a new message header and extensions to the
>    existing "received" message header are described.
>
> Under differerent circumstances I probably would have just brought this
> document up for discussion on this list. However, timing constraints are
such
> that Ted Hardie, my co-AD, suggested, and I agreed, that moving forward
with a
> last call at this time is the right thing to do. As such, I have requested
the
> IETF Secretariat to last call the document. (The last call announcent
> hasn't been sent yet but presumably will be sent soon.)
>
> Let the discussion begin...
>
>     Ned
>
>

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Comments requested on draft-malamud-no-soliciting-04.txt
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Date: Sun, 25 Jan 2004 08:12:35 +0100
Sender: [email-masked]

[email-masked] writes:
> Let the discussion begin...

It strikes me that many senders will want to pick at most one category, 
which requires receivers to add the entire IANA registry in the SMTP 
server configuration in order to avoid spam. That's not fortunate.

Arnt

Date: Sat, 24 Jan 2004 15:14:12 -0800 (PST)
From: [email-masked]
Subject: Comments requested on draft-malamud-no-soliciting-04.txt
To: [email-masked], [email-masked]
Cc: [email-masked], [email-masked]
Sender: [email-masked]

Folks,

A couple of weeks ago I received a request from the author to progress
draft-malamud-no-soliciting-04.txt to proposed standard. I reviewed the
document and sent a rather long list of corrections and suggestions to the
author, most of which have been incorporated into the -04 version.

Here's the document abstract:

   This Internet-Draft proposes an extension to SMTP for an electronic
   mail equivalent to the real-world "No Soliciting" sign. In addition
   to the service extension, a new message header and extensions to the
   existing "received" message header are described.

Under differerent circumstances I probably would have just brought this
document up for discussion on this list. However, timing constraints are such
that Ted Hardie, my co-AD, suggested, and I agreed, that moving forward with a
last call at this time is the right thing to do. As such, I have requested the
IETF Secretariat to last call the document. (The last call announcent
hasn't been sent yet but presumably will be sent soon.)

Let the discussion begin...

				Ned

Date: Sat, 24 Jan 2004 17:08:59 -0600
From: "Eric A. Hall" <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Nathaniel Borenstein <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

On 1/23/2004 6:40 PM, John C Klensin wrote:

> I would be happy, indeed delighted, to be wrong, but I see just 
> about no hope for designing and deploying a completely new email 
> structure.  More on that in draft-klensin-emailaddr-02.txt, 
> which I hope I can get wrapped up and posted on Monday or 
> Tuesday (it is a fairly major revision of the ideas in -01).

I disagree, but I'll read your draft when it's published anyway. From a
design perspective (or rather, ignoring the installed base as a criteria
of 'design', which is the usual and erroneous counter-argument), SMTP is
broken. The layering is munged and none of the advanced features will work
right until the transfer-headers and data-headers are clearly separated (I
know you're advocating a slight separation here, but that'll only get you
so far). Also, the messaging network is mostly limited to per-hop
negotiation, and does not provide the kind of end-to-end negotiation
(which is still possible in a store-and-forward environment) that is going
to be necessary for advanced features.

I agree with most of the rest of your message. I do not think it is
possible to do that with the existing infrastructure.

http://www.ehsco.com/misc/mt2/ has an outline of a replacement service
that covers most of the same ground and goes a fair bit further, the
latter of which is afforded by the fact that it's a wholesale replacement
[and no, Valdis, it doesn't require everybody to upgrade first :)]. I had
meant to get into I-D form last year but I've got too many other pro-bono
projects going already to take on another one. Steal as you wish.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 24 Jan 2004 13:35:24 -0800
To: (various IETF lists)
From: Paul Hoffman / IMC <[email-masked]>
Subject: New mail-ng mailing list open for sign-ups
Sender: [email-masked]

Greetings again. There seems to be more discussion these days about 
replacing SMTP and/or RFC 2822 and/or POP/IMAP for a variety of 
reasons. The discussion seems to pop up on a few different lists and 
in a few different hallways, and it might be good to have a single 
list where folks can congregate. Thus, I have set up a mail-ng 
mailing list.

The first task probably is to determine what the next generation of 
mail should do, and how that is different than what 
SMTP/2822/POP-or-IMAP or instant messaging does. It is safe to say 
that we can ignore actual protocol proposals for many months (if not 
years) until we figure out what is needed. Once we do that, there are 
plenty of protocol people who can attack the decided-on requirements.

There is no expectation that there will be significant agreement on 
the list. It is likely that over time the discussion will split into 
camps of like-minded design goals. The list might then spawn 
different lists for the folks of the different camps (mail-ng-shoe, 
mail-ng-sandal, ...). The list is explicitly not yet meant to be an 
IETF working group yet (if at all), and is probably more akin to the 
IRTF. But at the beginning, it will most likely be talking, not 
research.

See <http://www.imc.org/mail-ng/> for information on how to 
subscribe. The list is taking subscriptions now, and will probably go 
live for discussions within a week. Having some discussion on a 
mailing list now should make the dinners and bar gatherings at Seoul 
more interesting.

--Paul Hoffman, Director
--Internet Mail Consortium

Date: Sat, 24 Jan 2004 15:49:19 -0500
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Cc: [email-masked]
To: John C Klensin <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

John -- I'm glad to hear that you see this as part of a larger effort.  
I am certainly not opposed to incrementalism wherever possible, it just 
wasn't clear to me from this particular document, out of context, how 
it fit.

For my part, I think I might slice the problem a bit differently, but I 
might be wrong.  So I'd like to see our efforts start with some 
requirements analysis.  If we're going to build email NG, we need to 
make some decisions about what the goals are and then try to cut it 
off, rather than let it grow indefinitely as we're trying to make 
progress.

I've already suggested the following key goals:

	-- internationalization, esp. of addresses
	-- enhanced tracing mechanisms
	-- binary transport  (But what I really mean by binary transport is 
probably "deprecation of historical cruft," e.g. 
Content-Transfer-Encoding shouldn't be necessary where the extensions 
are used.)

I also think that there's enormous potential in generalizing and 
standardizing the kind of structured local-parts that people currently 
use "+" for.  And on and on.  The list of possible additions is fairly 
large, I'd like to have at least a rough consensus before we put a lot 
of work into documents.  -- Nathaniel

On Friday, January 23, 2004, at 07:40  PM, John C Klensin wrote:

> Nathaniel,
>
> We may be less in disagreement than you assume.  Briefly, I think it 
> is useful to write some things up separately, because, that way, it is 
> easier to think about whether the pieces are right or there are better 
> ways to do them.   And I am not punting on internationalization at all 
> -- it is my main concern here and I just want to see if we can build a 
> healthy environment for it, one in which, as I've said a few times, we 
> move aggressively toward an internationalized/multilingual Internet 
> rather than continuing with a predominantly Roman character/ English 
> network with various kludges, patches, and warts to permit other 
> scripts and languages to sort of work. So, fwiw...
>
> I would be happy, indeed delighted, to be wrong, but I see just about 
> no hope for designing and deploying a completely new email structure.  
> More on that in draft-klensin-emailaddr-02.txt, which I hope I can get 
> wrapped up and posted on Monday or Tuesday (it is a fairly major 
> revision of the ideas in -01).
>
> The nature of the internationalization problem, as I see it, is 
> precisely something that will need a number of interdependent changes. 
>  To try to make them independently, or to work around some of them, or 
> to try for small incremental chunks, is likely to get us into the land 
> of "nasty kludges forever".  I don't see that as flying, and I don't 
> see it as being successful.  On the other hand, if we look at the 
> whole picture (and if I'm right), when we get through, we have a 
> rather different-looking email environment, but one that is built on 
> the same general framework (avoiding hitting the wall of deploying 
> something completely different).  I see that picture as including:
>
> 	* UTF-8 addressing in both the local and domain parts
> 	(IDNA belongs in the application-DNS interface, not in
> 	the user-application interface).
> 	
> 	* handling of trace information in a way that doesn't
> 	screw up message bodies.
> 	
> 	* probably a fundamental rethinking of the trace
> 	information: While the draft hints at that, it isn't a
> 	can of worms I wanted to open in it.  But Received is,
> 	at best, badly outdated.  E.g., (i) more structure may
> 	be in order, (ii) it is a bad sign when we have to hide
> 	important information in comments, (iii) Via/With/ID
> 	need rethinking, (iv) it might make sense to sign the
> 	things or make assertions about authenticity or
> 	authorization, (v) the whole environment may need to be
> 	reconsidered in the light of common practices involving
> 	internal and external mail hosts,... and so on.  And, of
> 	course, the reason I wanted to call that command
> 	"envelope" and not "trace" is that, if an MTA is going
> 	to make an assertion that the message being transported
> 	is somehow virtuous and in a state of grace, maybe that
> 	shouldn't be done by tampering with the headers either.
> 	
> 	* UTF-8 (or equivalent) headers
> 	
> 	* probably a requirement for 8BitMIME, finally
> 	permitting this "next generation" environment to walk
> 	away from the 7bit environment.
>
> So my goal is to clean up our most problematic technical warts, move 
> forward aggressively with true internationalization, and to provide 
> hooks for better originator-based, authenticatable assertion-based, 
> permission-based, and/or authorization-based control of undesired 
> message flows (yes, in this context, I consider spam to be a subset of 
> a more general problem).   And that is, I think, pretty close to your 
> list of a "nice trio of goals".
>
> That said, my concerns about deployment issues and a good deal 
> --perhaps too much-- experience dealing with email problems leads me 
> to be very hesitant to take two (perhaps obvious) extra steps that I 
> think you are implying in passing:
>
> (1) Going to true binary transport, rather than continuing with a 
> line-oriented arrangement like 8BitMIME.  We pretty much know how to 
> do it; I'm just concerned about the interoperability and debugging 
> nightmares.
>
> (2) Pulling all of the headers out, so that "the envelope" becomes:
>
> 	Outer envelope:  Handshake and addressing information
> 	
> 	Middle envelope: Transport tracing, validation, and
> 	similar information.
> 	
> 	Inner envelope: More or less the semantic content of the
> 	2822-defined headers that are not part of MIME
>
> That way, the message payload more or less starts with a content type 
> and goes from there... or one could try to abstract MIME structuring 
> into an envelope layer too.  I could be wrong, but I just don't think 
> the complexity and transition problems -- especially for gateways-- 
> would be worth the costs.  But, if someone feels differently, I look 
> forward to seeing the drafts.
>
>      john
>
> 	
> --On Friday, 23 January, 2004 17:52 -0500 Nathaniel Borenstein 
> <[email-masked]> wrote:
>
>> John -- It's a pity to say this about such a nicely-written
>> draft, but I fear this proposal comes remarkably close to
>> maximizing the cost/benefit ratio.  When you consider
>> distributed costs of testing and deployment, I would bet that
>> implementing this proposal would cost at *least* 50% of what
>> it would cost to deploy a far more radical set of changes to
>> the email infrastructure.
>>
>> What we were discussing in Minneapolis was a
>> once-in-a-generation scale of protocol reform.  I think we
>> should think big, because there is a large fixed cost to
>> pushing any structural change through the entire Internet
>> community.  I'm particularly concerned that you seem to be
>> punting on internationalization -- I can't see putting much
>> energy into *any* major reform that doesn't address the most
>> widely-perceived failing of the current system.
>>
>> At root, what I don't buy is the notion that, in this case, we
>> can take an incremental approach to radical change.  The
>> Wright Brothers didn't learn to fly by practicing the high
>> jump, and I don't think we're going to clean up email by
>> creating a new architectural entity (the extended envelope)
>> that only works in ASCII.
>>
>> Now, if the extended envelope could be done completely in
>> UTF-8, with a coevolutionary model for son-of-822 header
>> fields and binary body transport, now *that* would be a more
>> compelling story.  Delivery tracing, internationalization, and
>> binary transport would be a nice trio of goals for a
>> next-generation email system.  I can certainly think of a few
>> others, though.  -- Nathaniel
>>
>> On Friday, January 23, 2004, at 04:53  PM, John C Klensin
>> wrote:
>>
>>> The draft posted/described in the attached is a bizarre idea,
>>> partially to see if it is possible to consider a radical
>>> solution to  an increasingly troublesome problem, and
>>> partially to see if the  supportive comments about "Email NG"
>>> in Minneapolis were really  serious.
>>>
>>> I am not at all convinced that it is a _good_ idea, only
>>> that, if we  are talking about radical changes to the mail
>>> infrastructure to  support various extended services, this is
>>> the sort of "clean up the  warts that get in the way" option
>>> we might want to consider.
>>>
>>> And even if it were a good idea, some of the details are
>>> probably not  right -- if this looks like it received a day's
>>> thought, you would  probably be guessing much too high.
>>>
>>> Discussion should probably go to the SMTP list; IMAA is
>>> copied only  because this could interact a bit with some of
>>> the "UTF-8 header"  discussions.
>>>
>>>   john
>>>
>>> From: [email-masked]
>>> Date: Fri Jan 23, 2004  3:59:11  PM America/Detroit
>>> To: IETF-Announce: ;
>>> Subject: I-D ACTION:draft-klensin-email-envelope-00.txt
>>> Reply-To: [email-masked]
>>>
>>>
>>> A New Internet-Draft is available from the on-line
>>> Internet-Drafts  directories.
>>>
>>>
>>> 	Title		: A Cleaner SMTP Envelope for Internet Mail
>>> 	Author(s)	: J. Klensin
>>> 	Filename	: draft-klensin-email-envelope-00.txt
>>> 	Pages		: 0
>>> 	Date		: 2004-1-23
>>> 	
>>> During the last few years, a number of proposals for
>>> extensions or improvements to email have run into trouble
>>> with a side-effect of mail relaying.  In the current Internet
>>> Mail model,  every SMTP server is required to break strict
>>> layering by inserting  one or more additional 'trace' headers
>>> into the message headers which  are actually part of the SMTP
>>> payload.  Since the headers are altered in transit,
>>> header-signing is not an available option, various anti-spam
>>> and internationalization strategies are infeasible or much
>>> more complex, and so on.  This document proposes to change the
>>> Internet mail model to place the in-transit trace information
>>> in the envelope, removing the requirement that relaying
>>> systems modify the message payload.
>>>
>>> A URL for this Internet-Draft is:
>>> http://www.ietf.org/internet-drafts/draft-klensin-email-envel
>>> ope-00.txt
>>>
>>> To remove yourself from the IETF Announcement list, send a
>>> message to ietf-announce-request with the word unsubscribe in
>>> the body of the  message.
>>>
>>> Internet-Drafts are also available by anonymous FTP. Login
>>> with the  username
>>> "anonymous" and a password of your e-mail address. After
>>> logging in, type "cd internet-drafts" and then
>>> 	"get draft-klensin-email-envelope-00.txt".
>>>
>>> A list of Internet-Drafts directories can be found in
>>> http://www.ietf.org/shadow.html
>>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>>
>>>
>>> Internet-Drafts can also be obtained by e-mail.
>>>
>>> Send a message to:
>>> 	[email-masked].
>>> In the body type:
>>> 	"FILE /internet-drafts/draft-klensin-email-envelope-00.txt".
>>> 	
>>> NOTE:	The mail server at ietf.org can return the document in
>>> 	MIME-encoded form by using the "mpack" utility.  To use this
>>> 	feature, insert the command "ENCODING mime" before the "FILE"
>>> 	command.  To decode the response(s), you will need "munpack"
>>> 	or a MIME-compliant mail reader.  Different MIME-compliant
>>> 	mail readers exhibit different behavior, especially when
>>> 	dealing with "multipart" MIME messages (i.e. documents which
>>> 	have been split up into multiple messages), so check your
>>> 	local documentation on how to manipulate these messages.
>>> 		
>>> 		
>>> Below is the data which will enable a MIME compliant mail
>>> reader implementation to automatically retrieve the ASCII
>>> version of the Internet-Draft.
>>> Content-Type: text/plain
>>> Content-ID:	<[email-masked]>
>>>
>>>
>
>
>
>
>
>

Date: Sat, 24 Jan 2004 08:31:45 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

--On Friday, 23 January, 2004 22:24 -0500 
[email-masked] wrote:

>...
> So for instance, the previous paragraph is why *any* proposal
> that doesn't provide benefit until 98% or so of the net has
> deployed it is doomed to fail unless you give a *really* good
> explanation of why 98% of the people should deploy it when it
> isn't going to give them any benefit if 3% of the OTHER people
> don't bother....

Valis,

I want to comment on the above, since I am in the process of 
proposing some things (the subject draft is only part of the 
package) that, from a deployment standpoint, give me the shakes 
-- and that should give anyone who has dealt with those issues, 
as you obviously have, similar anxieties.

At the antispam conference at MIT a week ago, there were an 
astonishing (to me) number of proposals that seemed to be based 
on the premise of "as soon as everyone on the network adopts 
this technology, everything will be wonderful".   And, as you 
suggest, the reaction among all of us with significant 
deployment experience was either "yes, right.  next?" or "when 
pigs fly".  It isn't even worth the effort to stand up and ask 
those folks pointed questions any more.  If you ever see a 
proposal with my name on it that proposes to cure spam (or the 
common cold) but requires Internet-wide deployment to be 
effective, you can safely assume that either the sender is an 
imposter or that I'm coming off a several-day drunk spell.

"If we managed to get this widely deployed ... it would help 
with ... or provide a framework for ..." is not that type of 
statement or proposal.  It is just an observation.  In that 
context, the _purpose_ of this proposal is to explore whether it 
is possible to remove (or start removing) a fairly serious wart 
in the way SMTP works without junking the infrastructure.  Would 
it be worth deploying if that were the only result?  Almost 
certainly not (at Nathaniel corrected pointed out).  Might it be 
worth deploying if it were part of a bundle of things that 
addressed an important problem (or group of problems) and 
cleaned some other things up?  Maybe, and that is one of the 
questions in front of us.

And, for precisely the reasons you identify, I want to explore 
what we really need from "mail NG" and what it would look like 
in the context of SMTP on Port 25.   Perhaps we could do better 
if we started with a clean slate.  But the costs --and risk of 
complete failure-- if we create a new mail infrastructure on a 
new port and try to transition to it are so formidable (even 
trying to work out the implications of DNS MX records boggles my 
mind) and frightening that it would be really attractive to 
avoid them.

But, it is equally important to note that there are some of 
these things that can be slipped in without requiring 
Internet-wide deployment to work and we should try to leverage 
them when we can.
We have looked at "Internet Fax", come to the conclusion that 
some of the needs of that community are a bit strange relative 
to the basic email infrastructure but that they _are_ a 
community, and gone ahead and specified things that make enough 
sense to them that they will deploy (and are deploying).  At 
least some of those extensions will eventually appear in 
mainstream SMTP MTAs, but many of us probably won't notice them 
when they do... and won't miss them until then.   One could make 
similar arguments for SMTP extensions that make sense within an 
enterprise, or otherwise between consenting adults, but maybe 
not on the public Internet (this is, after all, why we have 
X-commands).

Interestingly, one of the ways of looking at 
internationalization
at the header and envelope level is that it is more like fax 
than it is like "needs to be deployed across the whole Internet 
to make things work".  Remembering that the content part of the 
problem (can I send you something in a language that you don't 
read, have it arrive intact, and expect you to go out and find a 
translator) seems to have been solved a decade ago with the 
introduction of "content-type: text/plain; charset=...", and the 
potential for language tagging at the body part level, many 
years ago.  I've been surprised at how little I've seen 
multipart/alternative used with texts in different languages, 
but that seems to reinforce my present hypothesis.

The hypothesis is that the serious, crying, demand for i18n 
email addressing, and UTF-8 headers, etc., is communications 
within groups that share a language and script (primarily where 
one or the other isn't English/Roman-based).  Anything that is 
specified to enable those groups to communicate naturally 
intra-group had better still interoperate plausibly with the 
rest of the Internet, and with each other, or we kill email as a 
universal, worldwide, tool. But, just as we (implicitly) told 
those who where anxious to have multimedia facilities that they 
weren't going to get them in a convenient and interoperable 
environment until they --and their multimedia correspondents -- 
upgraded to MIME, I think it is plausible to say to the 
community that most wants i18n addressing,

	"This is a big change, as you know better than anyone
	else.  Nothing we do technologically is going to solve
	the problem of a user of Chinese communicating with a
	user of Hindi without understanding each other's
	languages or reading each other's scripts --especially
	for addressing, where the character strings in addresses
	may not be in either dictionary -- and, hence, you are
	likely to have to deal with those communications in some
	common language and/or script, probably (again, noticing
	world realities outside the Internet) English/
	Roman-based.   So we will make some extensions that will
	work well for communicating within your communities once
	you deploy them (and you have a _lot_ of incentive to
	deploy them) and that will preserve the same level and
	quality of communications that you have today with
	everyone else (no better, no worse) until the rest of us
	catch up... not just with the extension technology, but
	maybe even with learning to read and understand your
	script and/or language.
	
	"And, incidentally, the main alternatives, while they
	might deploy more quickly, are likely to lead to a lot
	of irritated users (who see internal codings because
	their environment wasn't upgraded and no one could tell)
	and/or information loss or other problems."

Now, is this argument correct?  I really don't know.  But it is 
one that I don't think we can reject out of hand, especially 
given the experience we have already accumulated when end users 
are told they have internationalization and local/native scripts 
and see internal and incomprehensible encodings in ASCII 
instead.   So I'm trying to explore it, see what it requires, 
and see if it leads into dead ends and what they are.

Scary?  Yes.  But I think it would be irresponsible to not take 
a hard look.

best,
      john

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt 
Date: Sat, 24 Jan 2004 05:58:16 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, January 24, 2004 3:36 AM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

> You're the one who brought up "closed mind attitude".

Its true! <g>

> You're also the one who asserted "You guys need to listen and begin
> opening your minds", without considering that maybe our minds *are*
> opened, and we're willing to listen to new ideas, but *no new ideas
> are forthcoming*.  More about that later.

In my view, the reason for this is that the functional specification is
inherently restrictive and this is what I meant
by "opening your minds" because maybe then some legit effort can begin.  I
don't think I am off based the general consensus viewpoint most often cited
about the IETF is that it is a closed knit group of vets where proposal for
change is philosophically, militantly and fundamentally not welcome. :-)

> to the people who may read the list archives in the future, trying
> to figure out why *their* brilliant pet proposal got shot down...

People will read this tone, the negative "brilliant pet proposal"
description in the archives and they will ASSUME a "closed minded attitude"
and become more resistance to participate because the tone ridicules them.

> OK, You tell me.  What do *you* tell one of your engineers when
> they come in with the *same* idea that 8 other engineers have
> suggested, and this latest one hasn't done anything about the
> fatal flaw you already pointed out to 8 others?

Since you ask :-)  Might be off base, you would probably say:

     "Be there, done that. Don't need to go there."

The fellow will be less encourage to provide future input because of your
unintentional, no harm intended, closed minded
intimidation.

Me?

I would listen first and not oppress creativity by presuming the idea is the
"same."  It may sound and inevitably when all things considered, be the
same, but I will not dampen and kill the realistic possibility this guy
might "see something" the other 8 did not.  In short, play dumb and allow
him to talk.

     "Interesting Tom.  You know some research was already done
     in this area. You have a slightly different twist in this
     particular area.  Why not run it by so and so...."

If your design research philosophy prevailed, technological innovation would
of been oppressed a long time ago, probably
explaining why SMTP is at where it is now - stalled!   It is not uncommon to
have different groups of engineers be asked to tackle the same problem with
the primary purpose of not allowing a singular thought process to prevail.

> Exactly.  Unfortunately, some of the "historical perspectives" the
> veterans have are:
>
> 0) Most of the following is oriented towards anti-spam stuff, mostly
because
> that's the single biggest unsolved problem in the SMTP world.

Agreed. So what are we doing about it?

>
> 1) Most "new ideas" are re-inventions of ideas already proposed anywhere
from
> months to decades before.

So what? Allow the possibility of a new possible view point.  You simply
never know.

>
> 2) Most of said ideas have a fatal flaw.  If they didn't, we'd probably
>    have deployed them the *first* time they were suggested.

And the fatality of these ideas are based on a flawed functional
specification. When the technical design criteria for any new SMTP
inventions must be backward compatible, all new ideas are inherently flawed.
When the specs has it written in stone that "it is flawed" from a security
standpoint, then makes no attempt to address it, all ideas are lost.  Why
bother?

This is the only issue I have with the IETF - this viewpoint needs to change
and by that I mean, once again, the "enforcement" *SHOULD* be allowed,
otherwise the idea is weak right out of the starting cage.

> 3) There's been a *lot* of ideas with fatal flaws.  See Vernon Schryver's
> intro to the subject: http://www.rhyolite.com/anti-spam/you-might-be.html
> And keep in mind that list is not exhaustive - it's merely the anti-spam
> ideas Vernon has seen *so* many times he got annoyed.

I've seen this in the past.  Silly, childish, again, all based on the
fundamental flaws of the specs prohibiting any new idea to
even possibly have a chance to be researched, be adopted or enforced!  All
ideas are flawed because none are technically enforceable.  Most ideas
addressing anti-spam are flawed because the specs has it written in stone
that any attempt to validate or control access is technically flawed and
btw, technical not compliant.  I don't know about you, but there is
something wrong with that picture.

Also, my approach to the anti-spam problem at the SMTP  level is not to
interpret mail-content but by applying new anonymous access control
restrictions at the protocol level independent of what the mail-content
might really be - good or bad, if the sender is not "compliant," no
transaction - period.

>
> 4) There's still spam.  Why? Because none of *us* are clever enough
> to come up with an idea that doesn't get nailed by at least one point
> on Vernon's list either. As a result, the most productive use of our
> time is to point everybody at that list or some similar, and have them
> self-evaluate their proposal, and let us know if they think they're
> really on to something new.

Again, you point out what I am talking about. Assuming Vernon's list has any
validity, why is failing 1 point considered a complete failure? And again,
its all based on the fundamental flaws of the specs. So until you see this,
there is NO going to be any progress whatsoever with IETF vets. <g>

Not to spur up another debate, I don't have a spam problem any more. That
problem has been solved. Yet, I might see 1 spam might sneak in out of 3000+
attempts per day.  I think that is pretty good.  I don't see that as a
failure, and yes, I am using a CBV as the heart of the suite of anti-spam
methods using a basic PRESUMPTION the return path must be verifiable. So for
 us, SPAM is a thing of the past.

If you disagree with that presumption, and are not willing to see that there
is a conflict with CAN-SPAM which mandates it must be valid and that the
spammers exploit this to the maximum extent, then maybe you can see why I
say you need to open your minds more.

If you disagree with the CBV concept for scalability issues, and are not
willing to "research" an optimal validation concept
(which implies you first need to agree that it is enforceable), then again,
maybe you can see why I say you need to open your minds more.

The problem is not the SMTP protocol. The problem is the functional
specification. It conflicts with the realities of
today's industry problems. It restrict innovation. It restricts enforcement.
It is the REASON the industry problems exist in the
first place to the extent of a $8 Billion dollar industry wide price tag.

What allows this happen?  The smtp protocol or the functional specifications
that prohibits innovation?

Spammers love you because it is the latter that allows it to happen. They
are laughing silly all the way to the bank
exploiting and abusing our networks and machines at the same time.

> 5) Never try to read too much into the postings of anybody
> who posts at 3:30AM who isn't actually employed as third-shift
>  personnel. :)

Point well taken. <g>

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt 
From: [email-masked]
Date: Sat, 24 Jan 2004 03:36:03 -0500
Sender: [email-masked]

--==_Exmh_-2119474378P
Content-Type: text/plain; charset=us-ascii

On Sat, 24 Jan 2004 02:16:34 EST, Hector Santos said:

> May I ask, why do you assume I lack system design or system wide engineering
> integration, design impact, understanding , capability or experience?

I didn't.

You're the one who brought up "closed mind attitude".  You're also the one
who asserted "You guys need to listen and begin opening your minds", without
considering that maybe our minds *are* opened, and we're willing to listen
to new ideas, but *no new ideas are forthcoming*.  More about that later.

You're also making the rash assumption that my reply was directed entirely to
you personally, and not to the lurkers on the list, or to the people who may
read the list archives in the future, trying to figure out why *their*
brilliant pet proposal got shot down...

> 1) the flawed assumption a proposed consideration, concept or idea is not a
> good idea to begin with at any level minimizing possibility or consideration
> for implementation, and

OK, You tell me.  What do *you* tell one of your engineers when they come in
with the *same* idea that 8 other engineers have suggested, and this latest one
hasn't done anything about the fatal flaw you already pointed out to 8 others?

> All I am saying it is a mistake I lack reason,  knowledge or insight in my
> statements.  I appreciate the experience and historical IETF perspectives
> shared by your and other IETF veterans here .  But I will not allow views I

Exactly.  Unfortunately, some of the "historical perspectives" the veterans have
are:

0) Most of the following is oriented towards anti-spam stuff, mostly because
that's the single biggest unsolved problem in the SMTP world.

1) Most "new ideas" are re-inventions of ideas already proposed anywhere from
months to decades before.

2) Most of said ideas have a fatal flaw.  If they didn't, we'd probably have
deployed them the *first* time they were suggested.

3) There's been a *lot* of ideas with fatal flaws.  See Vernon Schryver's intro
to the subject: http://www.rhyolite.com/anti-spam/you-might-be.html And keep in
mind that list is not exhaustive - it's merely the anti-spam ideas Vernon has
seen *so* many times he got annoyed.

4) There's still spam.  Why? Because none of *us* are clever enough to come up
with an idea that doesn't get nailed by at least one point on Vernon's list either.
As a result, the most productive use of our time is to point everybody at that list
or some similar, and have them self-evaluate their proposal, and let us know if
they think they're really on to something new.  

5) Never try to read too much into the postings of anybody who posts at 3:30AM
who isn't actually employed as third-shift personnel. :)

--==_Exmh_-2119474378P
Content-Type: application/pgp-signature

--==_Exmh_-2119474378P--

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Sat, 24 Jan 2004 02:17:48 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

>
> That is very encouraging, although I would encourage you to read
> and understand the proposal, and probably to follow the
> discussion that is likely to unfold on this list (and, for part
> of the work, the ietf-imaa one) before committing yourself to
> implement anything.
>

John,

Please yield to any lingering benefits of your doubts you may have about me.
I am not an idiot.

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt 
Date: Sat, 24 Jan 2004 02:16:34 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, January 23, 2004 10:24 PM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

> What is often mistaken as "old guru closed mind attitude" is the
> experience to be able to tell that *any* proposal in category X
> is doomed to failure unless problems A, B, and C are addressed,
> and any proposal in Y has almost-certain fatal flaws D and E, and
> so on.

May I ask, why do you assume I lack system design or system wide engineering
integration, design impact, understanding , capability or experience?

What is becoming quite clear to me, it seem to have a fundamentally flawed
assumption about me.  I am trying to be nice.  I do not make the assumption
you or anyone else here is not competent.

> Another common error is failing to understand the realities of
> scaling and deployment.

Yet another fundamentally flawed assumption. Why do you assume I lack basic
understanding in high end product operations?

> It would take me a short afternoon to put
> something up to test, understanding that it was beta-quality
> software.

Again, why do you presume I lack this understanding? Do you an exclusive
copyright on this concept?

> Getting everybody in my department moved is a much
> bigger challenge, and migrating 60K users is a year-long
> proposition, and migrating the millions of users of an AOL or
> Hotmail is a daunting issue indeed. (Hint - how many extra
> first-line help desk people would AOL need to hire, and what
> would the price be for that?

Sure, nothing but a sound migration plan can't address.  But again, why do
you assume I don't understand this?  I said nothing to indicate otherwise.

> Is the cost of the migration more than their current anti-spam costs, or
not?

Maybe, maybe not.  But again, was there anything specific I said to suggest
I don't understand this concept?

> Don't forget to include burning 100M new coaster^H^H^H^H^H CD's with 'AOL
10.0 -
> now with new anti-spam' on it).

Again,  what was it did I said to suggest I don't understand this
distribution cost?

> TV ads.  Lots of them.  You'll need them.  If you don't
> understand why, don't bother commenting until you do. ;)

or that I don't understand product marketing?

> Oh, and don't forget that you're not allowed to break anybody's
> connectivity at any point along the line.  The NCP->IP migration
> was bad enough wnen there were only several hundred hosts.

One bad experience should not block your vision.

> We'll never get to do that again.  If you have half the users migrated
> to SMTP-bis and half are still on SMTP-classic, they *will* have
> to be able to communicate, or you *will* be fired.

I'm the boss so it can't happen! <g>  However, I don't mind saying this
tangential ad nauseum reading is something I look for when considering a new
job applicant.  :-)  Having insight and hindsight is one thing. Allowing it
block progress or vision including the discussion of the possibility is
another.

> So for instance, the previous paragraph is why *any* proposal
> that doesn't provide benefit until 98% or so of the net has
> deployed it is doomed to fail unless you give a *really* good
> explanation of why 98% of the people should deploy it when it
> isn't going to give them any benefit if 3% of the OTHER people
> don't bother....

Well,  I used pareto's principle (look it up) when it comes to most aspects
in my natural, everyday decision making process.  Most successful people do
and with great success in the end result.

With that said,  I believe there are some fundamental flawed assumptions in
your particular statement above:

1) the flawed assumption a proposed consideration, concept or idea is not a
good idea to begin with at any level minimizing possibility or consideration
for implementation, and

2) the flawed assumption a proposed consideration, concept or idea is not a
well written and does not include a sound acceptance and deployment plan,
i.e., the author did a terrible job,  and

3)  the fundamental flawed assumption that a proposed consideration, concept
or idea will not be (cost) effective or beneficial unless it is deployed by
a majority, and

4) finally, the fundamental flawed assumption that a proposed consideration,
concept or idea can not be fine tuned over time to meet whatever benefits
sought to maximize acceptability and deployment.

[SOAPBOX]

FWIW,  I honestly hate stooping to this level,  but I feel you have the
wrong impressive about me.  I am getting the impression that you (and
others) honestly think I'm a "dummy,"  lack product design, development and
deployment experience and insight  and overall feel there is no merit or
reason in my statements.  So excuse me if I may provide you a better insight
about me. I am a chemical engineer by trade with 28+ years software/product
design/engineering/production experience that will make most people puke.
Since 84, I have been very active in designing, development and supporting
mail based products.  I helped pioneered the off-line mail market, certainly
the first to mass market the concept during these early days.   My company
currently serves a relatively small market of over 100,000 customers using
one or more of our intranet, dial-up, and/or file/mail server hosting
product lines.  Sure, probably small potatoes in your mind, but its big
enough for us mind you. It keeps us busy and more importantly, debt free and
profitably.  I like to think our competitive advantage is superior design
and responsive customer support.  My long standing strong ethical
engineering principle and strategy for the company is simple: Minimize your
bugs, offer robust, fast, no compromising technical design concepts and your
customer satisfaction is maximized and your customer support issues,
overhead and cost is minimized.  As a small company, this is a must.   We
are #1 in a particular fortune 500 market niche offering a sound time-tested
engineered integrated client/server design framework.   Yes,  SMTP (the mail
system) is not our main niche, but its among our highest growth areas.
Based on shared former Lotus Notes and MS Exchange customer testimonies,
our mail system is considered very worthy, very scalable and without a
doubt, definitely considered the fastest.   Our mail server which supports
various gateways such as Fidonet (Netmail and Echo Mail), Internet (Email
and News) and other mail networks such as QWK/OPX/RIME networks was
originally designed from the ground up for high volume satellite based feeds
and ISP mail hosting    Processing 100-155 messages/sec was a design
requirement.  Yada, yada, yada, yada...   again I'm embarrassed to stoop to
this need, but you make sound like I am some "young punk." <g>.

All I am saying it is a mistake I lack reason,  knowledge or insight in my
statements.  I appreciate the experience and historical IETF perspectives
shared by your and other IETF veterans here .  But I will not allow views I
may not agree with block my own creativity in problem solving. It has worked
for me for many years.  I came here with an open mind with a desire to have
a solid professional technical discussion that improves or addresses the
major current industry issues we are facing.   Honestly, I would love to
meet the authors of all the SMTP packages.   So yes, I don't mind saying
maybe I have a selfish focus here. I am a commercial developer which I
sincerely hope you and others here do not hold against me.  This focus
should not suggest to you I lack insight when it comes to new ideas..  I am
100% aware and understand design implementation issues and yes,  I
understand resistance for change.   I agree 100% that unless an SMTP
altering idea has a major benefit,  it *MAY* not (not MUST not) be worth
considering.  My goal is to help if I can and that includes being open
minded.

[/SOAPBOX]

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt 
From: [email-masked]
Date: Fri, 23 Jan 2004 22:24:39 -0500
Sender: [email-masked]

--==_Exmh_362543613P
Content-Type: text/plain; charset=us-ascii

On Fri, 23 Jan 2004 21:21:15 EST, Hector Santos said:
> mindset.  Quite frankly,  there seems to be conflictive old guru closed
> minded attitude about the realities of the mail industry delimas which
> perpetuates a dearth of interest to finally address the issues one way or
> another.    You guys (speaking in general) need to listen and begin "opening
> your minds."

What is often mistaken as "old guru closed mind attitude" is the experience
to be able to tell that *any* proposal in category X is doomed to failure unless
problems A, B, and C are addressed, and any proposal in Y has almost-certain
fatal flaws D and E, and so on.

Another common error is failing to understand the realities of scaling and deployment.
It would take me a short afternoon to put something up to test, understanding that
it was beta-quality software.  Getting everybody in my department moved is a
much bigger challenge, and migrating 60K users is a year-long proposition, and
migrating the millions of users of an AOL or Hotmail is a daunting issue indeed.
(Hint - how many extra first-line help desk people would AOL need to hire, and
what would the price be for that?  Is the cost of the migration more than their
current anti-spam costs, or not?  Don't forget to include burning 100M new
coaster^H^H^H^H^H CD's with 'AOL 10.0 - now with new anti-spam' on it).

TV ads.  Lots of them.  You'll need them.  If you don't understand why,
don't bother commenting until you do. ;)

Oh, and don't forget that you're not allowed to break anybody's connectivity at any
point along the line.  The NCP->IP migration was bad enough wnen there were only
several hundred hosts.  We'll never get to do that again.  If you have half the users
migrated to SMTP-bis and half are still on SMTP-classic, they *will* have to
be able to communicate, or you *will* be fired.

So for instance, the previous paragraph is why *any* proposal that doesn't provide
benefit until 98% or so of the net has deployed it is doomed to fail unless you give
a *really* good explanation of why 98% of the people should deploy it when it isn't
going to give them any benefit if 3% of the OTHER people don't bother....

--==_Exmh_362543613P
Content-Type: application/pgp-signature

--==_Exmh_362543613P--

To: [email-masked]
From: [email-masked]
Date: Fri, 23 Jan 2004 22:00:01 -0500
Sender: [email-masked]

--==_Exmh_357094683P
Content-Type: text/plain; charset=us-ascii

On Fri, 23 Jan 2004 20:23:31 EST, [email-masked]  said:

> "client/server protocol compliant level" identification.  SMTP should be =
> given a "IETF VERSION" stamp, much like HTTP 1.0/2.0, etc. and it should =
> be part of the protocol handshake (or possibly DNS MX lookup) that =
> clients and servers can use to determine level of compliancy.  While =
> ELHO can be used to identify extensions,  deployment is minimized by not =
> having an enforcement statement in the functional specifications.  =

The problem with a "version stamp" is that it imposes a strict partial ordering
on features.  For example, if PIPELINE was included in SMTP 1.5, and SIZE was a
SMTP 1.8 feature, there's no clear way to implement SIZE without doing PIPELINE
as well.  This can really be a headache if you have a resource-constrained
target (think "cell phone") where you *really* need SIZE so you can control
bandwidth usage, but you don't have the capacity to do PIPELINE (which is a lot
harder to get right than it looks at first glance).

You should go back in the old MIME working group discussions and read up on why
it wasn't believed we'd every have a 'Mime-Version: 2.0' header...

--==_Exmh_357094683P
Content-Type: application/pgp-signature

--==_Exmh_357094683P--

Date: Fri, 23 Jan 2004 21:36:02 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

Replying to ietf-smtp only...

Hector,

As I indicated in my response to Nathaniel, this proposal is one 
of a group, most of whole elements are still to come (and some 
of the elements of which are small variations of work already 
done by others).  Also, the draft assumes that one has read 
carefully and understood 2821 and its terminology.  If this goes 
anywhere --and the purpose for posting it and some of its 
pending relatives is to start some discussion, not necessarily 
to have this result come out-- then it may be desirable to make 
it more self-contained... or it may make better sense to combine 
it with some other proposals.

I'll respond to a few of your other questions/ comments below.

--On Friday, 23 January, 2004 20:03 -0500 Hector Santos 
<[email-masked]> wrote:

> In general,  I have much to say about any SMTP proposal that
> suggests SMTP change at the software level. To me, a change in
> software opens the door for other possible concepts that may
> address a particular needs in the email industry.   IMTV,
> acceptance and deployment aspects of the proposal are
> important and major considerations of any SMTP change
> proposal.  IMTV, that's a matter of a having a strong
> functional specification dictation.
>
> With that said,  I am putting my "Software Engineering Hat" on
> as if I was going to implement your ENVL proposal today to see
> what would be the technical implementations issues.  Based on
> this technical point of view, here are my specific comments
> about your draft:
>
> What does this ENVL proposal attempt to address?  (What
> problem(s) does it solve?)

Mentioned in the text, and discussed many times, in different 
contexts, over the last decade or two: the narrow purpose of 
this proposal is to separate the "trace" information from the 
[rest of] the header information.   The impact of doing that is 
that in-transit (including relay) MTAs no longer have to inspect 
or take any other action with regard to (other) message headers. 
Looked at from the other end, that also implies that the headers 
as they leave the originating MUA (or the combination of the 
originating MUA and the submission MTA-- see 2821) can be (and 
can be required to be) identical to the headers as they reach 
the delivery MTA and, local post-SMTP-delivery decisions aside, 
the message store and receiving MUA.  That has a number of 
potentially useful implications, which, again, are mentioned in 
the text.

> Excuse me if I missed it in the draft, but I see only three
> (3) concepts that addresses a "problem is solved:"
>
> 1) Proposal provides a cleaner SMTP envelope or proposal to
> create a cleaner RFC 822 trace header?

This proposal moves the 822 (actually [2[821) trace headers from 
the headers (i.e., part of the message body / payload) into the 
SMTP envelope. By doing so, it provides what I think of as a 
cleaner envelope because, today, SMTP servers work with both the 
envelope (as defined in SMTP transactions) and also with the 
headers (because they need to push the trace information into 
them).  See Nathaniel's notes and my response to them.

> 2) Cleaner SMTP envelope minimizes mail relay issues.

See above and the draft.

> 3) Syntax Error checking.

Optional.

> Is this correct?

See above, Nathaniel's notes, and my response.

> What is it about the headers today that makes it problematic?

See above and the draft.

> You touch base with section 3, item 6 indicating "Syntax
> Error."   Syntax Error needs to be clarified, i.e,  can (or
> may) this include an implementation that attempts to perform
> some sort of sender/machine validation?

Unless information additional to that which 2821 requires/ 
permits in Received is provided somehow, there is no meaningful 
and reliable way to do such validation, especially using 
envelope (plus Received field) information only.  See my 
response to Nathaniel and, if you think such validation would be 
useful, please start writing --not email but an internet-draft.

> What is considered the ENVL header?

It is not a header, it is, under this proposal, an SMTP command.

> The term "Trace Fields" are used.  This needs to be clarified
> for the software engineer who may not be in the same league as
> protocol police or IETF gurus.  It seems to indicate only the
> Received: and Return-Path: lines are considered and are the
> ENVL transaction.

Please see the section called "Trace Information" (4.4, I think) 
in RFC 2821.  Also please read section 5, item 2, of the posted 
draft -- it is really very clear about this.

<tirade>
Much more generally, "software engineers" and others who choose 
to start implementing -- or even to start commenting on -- 
documents they haven't taken the time to read and make a serious 
attempt to understand are a menace to themselves and to their 
employers (if any) and users.  Many times, a first-draft 
proposed protocol specification is not as clear as it should be 
(and certainly mine are often not): that is one of the many 
reasons we post drafts for comment and review and why both the 
IESG and the RFC Editor review them (at least the 
standards-track subset) for clarity.   But, in this case, I 
deliberately used the 2821 (and 821 and earlier) terminology of 
"trace fields" (or "trace information") rather than saying 
"Received: and Return-path:" and then I wrote a paragraph 
explaining why I did that.  Expecting people to read, and keep 
track of, every word and provision would probably be 
unreasonable if this draft were the length of 2821, but, 
exclusive of boilerplate and structural elements, this one is 
only about 4 1/2 pages long -- and trying to keep it short was 
precisely the reason I didn't bundle this proposal with others 
in the group (again, see the discussion with Nathaniel).   It 
really does no one any good if someone has to repeat the 
comments of an I-D on the mailing list (and, often, again in an 
WG or BOF session) because someone has reacted to it without 
reading it, whether the person who isn't reading the text 
identifies him or herself as a "software engineer", an "IETF 
guru", or something else.
</tirade>

> Lets ask this another way:
>
> Does this proposal basically attempts to separate the
> transmission of a RFC 822 formatted message normally done at
> the DATA state into two separate state transactions?   For
> example:
>
>     C: ENVL
>     S: 250 Start Mail Header input; end with <CRLF>.<CRLF>
>     C: [sends header followed by "." line]
>     S: 250 Mail Header OK. Continue with Data Command
>
>     C: DATA
>     S: 354 Start Mail Body input; end with <CRLF>.<CRLF>
>     C: [send message body only (no header) followed by "."
> line]     S: 250 Message accepted for delivery.

First of all, the text is very explicit (section 3 (5)) that 
there is no "good ahead" handshake after the ENVL command.  It 
also specifies (same section) that what is sends is _only_ trace 
field information, not miscellaneous headers.

> or are we just adding a new state that allows a client to
> provide a server with access to "DATA" header (all or
> specific) information using the ENVL before the actual DATA
> transaction takes place?

Since I don't know what a "DATA header" is, I have no idea what 
you are asking about.

>     C: ENVL
>     S: 250 Start Mail (Trace??) Header input; end with
> <CRLF>.<CRLF>     C: [sends mail header (all or just some
> specific headers??) followed by "." line]
>     S: 250 Mail Header OK. Continue with Data Command
>
>     C: DATA
>     S: 354 Start Mail Header/Body input; end with <CRLF>.<CRLF>
>     C: [sends mail header again and body followed by "." line]
>     S: 250 Message accepted for delivery.

See above.

> In short,  an example for the "laymen" would be nice :-)
>
> Anyway, I personally believe your ENVL is a good start.  It
> has promise. Off hand, without grasping yet the true intent
> this proposal (what does it solve),  what defines the ENVL
> "headers", etc,  I believe the answer to this will have a
> major factor on the acceptance and deployment aspect of the
> proposal.  Besides other impact considerations which I don't
> wish to muddy the water just yet without getting the answer to
> the above,  if this ENVL or another other SMTP change proposal
> proves to be useful,  I'll be among the first to implement
> such a new SMTP feature like this in our SMTP product   I will
> be interested on providing an immediate  "proof of concept"
> wide test bed of customers.

That is very encouraging, although I would encourage you to read 
and understand the proposal, and probably to follow the 
discussion that is likely to unfold on this list (and, for part 
of the work, the ietf-imaa one) before committing yourself to 
implement anything.

best,
    john

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Fri, 23 Jan 2004 21:23:25 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I agree Nathaniel.  Any change to SMTP needs to be significant in what it
addresses and if done, the opportunity should be use to address ALL major
contemporary mail industry issues to minimize future change or extension
requirements.

In addition, I know this may not be a popular view,  I don't believe any
change (or extension) would have significant impact unless the functional
specifications make it possible NOT to be BACKWARD compatible at the system
admin policy level. In other words, allow the SysAdmin to decide the level
of compliance. In this vain, I believe that future SMTP change proposals
should begin with a design review on the concept of "client/server protocol
compliant level" identification.  SMTP should be given a "IETF VERSION"
stamp, much like HTTP 1.0/2.0, etc. and it should be part of the protocol
handshake (or possibly DNS MX lookup) that clients and servers can use to
determine level of compliancy.  While ELHO can be used to identify
extensions,  deployment is minimized by not having an enforcement statement
in the functional specifications.  Future SMTP operations require stronger
policy base operations so that if a COMPANY does not want to allow for
example, non-ENVL compliant clients, it should be allowed because the
functional specification (RFC) allows developers to make it possible.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: Nathaniel Borenstein
To: John C Klensin
Cc: [email-masked]
Sent: Friday, January 23, 2004 5:52 PM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

John -- It's a pity to say this about such a nicely-written draft, but I
fear this proposal comes remarkably close to maximizing the cost/benefit
ratio. When you consider distributed costs of testing and deployment, I
would bet that implementing this proposal would cost at *least* 50% of what
it would cost to deploy a far more radical set of changes to the email
infrastructure.

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>, "Nathaniel Borenstein" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Fri, 23 Jan 2004 21:21:15 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Nathaniel Borenstein" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, January 23, 2004 7:40 PM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

>
> Nathaniel,
>
> We may be less in disagreement than you assume.  Briefly, I
> think it is useful to write some things up separately, because,
> that way, it is easier to think about whether the pieces are
> right or there are better ways to do them.   And I am not
> punting on internationalization at all -- it is my main concern
> here and I just want to see if we can build a healthy
> environment for it, one in which, as I've said a few times, we
> move aggressively toward an internationalized/multilingual
> Internet rather than continuing with a predominantly Roman
> character/ English network with various kludges, patches, and
> warts to permit other scripts and languages to sort of work.

Is this one of the main reasons for the ENVL proposal?
Internationalization?   if so, this sort of answers my question on what the
ENV header includes.  It implies mime content headers are part of the ENVL
header block.  No?

> So, fwiw...
>
> I would be happy, indeed delighted, to be wrong, but I see just
> about no hope for designing and deploying a completely new email
> structure.

You know, this pessimistic attitude is going to help anyone.   Fwiw,  while
I value the strong history and knowledge of SMTP/IETF gurus provide in the
forums,  what I find extremely disappointing is this "my way or highway"
mindset.  Quite frankly,  there seems to be conflictive old guru closed
minded attitude about the realities of the mail industry delimas which
perpetuates a dearth of interest to finally address the issues one way or
another.    You guys (speaking in general) need to listen and begin "opening
your minds."

With that said,  you don't need to justify why your initial draft isn't
perfect.  Its a start.    In my view, the case has been MADE for a "change
discussion" at a minimum and part of that discussion should be how to best
address backward compatibilty and to maximize acceptance and deployment.

As I mention in my previous message to Nathaniel, in my view, a change to
smtp will not have a major acceptance and deployment unless the IETF mindset
is changed on how the functional specifications are written to provide or
OFFER the possibility for stronger SMTP compliancy level system policies by
system administrators.   In my view, unless this is done, nothing will have
signfiicant impact because the specs says you don't have to follow it.
While I believe backward compatibilty is important, it should also be
allowed for developers begin to make it a possibility for offering stronger
system admin policies.

>  * handling of trace information in a way that doesn't
>  screw up message bodies.

In my view, this ENVL proposal can be simplified by basically making it an
alternative to constructing the RFC 822 message:

CURRENT:

    DATA ---.> RFC 822

ENVL PROPOSAL:

    ENVL + BODY (or DATA) ----> RFC 822

The point here is that ENVL can now provide developers  a way to technical
provide system policy power similar to what YAHOO is attempting doing today
using the current SMTP specification; Delay RCPT validation and verification
until it sees the headers in the DATA stage, in addition provide an
technical reason to make their YDK (Yahoo Domain Key) proposal a technical
possibility.   So ENVL will optimize performance for YAHOO and others who
use this for verification/validation and/or YDK implementation.

And more importantly, something like ENVL  can finally provide system policy
power for developers to address various mandates imposed by the US Federal
Law AKA the CAN-SPAM act without requiring the DATA transaction which is in
my view one of the CAN-SPAM loopholes spammers are patting themselves on the
back about. The CAN-SPAM Act requires the spammer to identify the topic of
the mail content.  This can't be done until the DATA is collected and for
spammers reaching the DATA stage, it it like seeing the "gold at the end of
the rainbow."

> Pulling all of the headers out, so that "the envelope"
> becomes:
>
>  Outer envelope:  Handshake and addressing information
>
>  Middle envelope: Transport tracing, validation, and
>  similar information.
>
>  Inner envelope: More or less the semantic content of the
>  2822-defined headers that are not part of MIME

I think this complicates the required solution at the SMTP level that
something like ENVL can help address.   To me, SMTP should not be in the
business of interpreting mail-content.  The presumption is RFC 822 compliant
ready data handled at both ends of the end-user mailer software with SMTP
network headers included in the process.   To me, the most important
industry issues are related to access control and mail transport integrity.
So in my view,  SMTP protocol change proposals should concentrate on
addressing these two very important and real concepts that will have a major
and direct effect on the industry concerns.

I like the ENVL proposal  if it allows for the SERVER to see/analyze RFC 822
header information before the DATA stage is reached.  To me, that is the
most important benefit it will bring to the industry.  Otherwise, I see no
benefit gained from it.

Thats my view.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Fri, 23 Jan 2004 18:25:00 -0800 (PST)
From: [email-masked]
To: [email-masked]
cc: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

The proposal does not really address "cleaner" or "more detailed" message 
routing data or define standards for it. What it does is to separate the 
actual message from the routing/envelope/trace data being added by mail 
servers while the message is in transit. Additional proposals would have 
to build on this one to define more trace data or more information about 
message routing, etc.

One possible benefit is that mail filtering systems would be able to deny 
message or bounce it if data is not acceptable to it (not properly signed, 
bad origin, etc). This has certain processing/bandwidth benefits as 
message would not have had to be processed or tranmitted in full (and then 
bounced back to "envelope-from" address which may not even exist) and 
could be bounced immediatly oe possibly certain authentication data could 
be requested and immediatly check on.

In general to other remarks I have to agree, it is not perfectly clear if 
its better to add number of additional extensions that may solve (close 
holes) in current email transport system or if its better to just design 
new mail transport protocol all together. However it maybe of benefit to 
everybody if we work on both approaches at the same time for now and 
decide what is better in the future (several prcidents to that - CRISP has 
worked on both FIRS/LDAP and IRIS/XML variants; LEMONADE is working on 
extensions for IMAP to do both do remote mail editing & submission and 
different approach to reference parts of imap message in message being 
submitted direct from end-user computer)

On Fri, 23 Jan 2004, Hector Santos wrote:

> 
> Hello John,
> 
> In general,  I have much to say about any SMTP proposal that suggests SMTP
> change at the software level. To me, a change in software opens the door for
> other possible concepts that may address a particular needs in the email
> industry.   IMTV, acceptance and deployment aspects of the proposal are
> important and major considerations of any SMTP change proposal.  IMTV,
> that's a matter of a having a strong functional specification dictation.
> 
> With that said,  I am putting my "Software Engineering Hat" on as if I was
> going to implement your ENVL proposal today to see what would be the
> technical implementations issues.  Based on this technical point of view,
> here are my specific comments about your draft:
> 
> What does this ENVL proposal attempt to address?  (What problem(s) does it
> solve?)
> 
> Excuse me if I missed it in the draft, but I see only three (3) concepts
> that addresses a "problem is solved:"
> 
> 1) Proposal provides a cleaner SMTP envelope or proposal to create a cleaner
> RFC 822 trace header?
> 
> 2) Cleaner SMTP envelope minimizes mail relay issues.
> 
> 3) Syntax Error checking.
> 
> Is this correct?
> 
> What is it about the headers today that makes it problematic?
> 
> You touch base with section 3, item 6 indicating "Syntax Error."   Syntax
> Error needs to be clarified, i.e,  can (or may) this include an
> implementation that attempts to perform some sort of sender/machine
> validation?
> 
> What is considered the ENVL header?
> 
> The term "Trace Fields" are used.  This needs to be clarified for the
> software engineer who may not be in the same league as protocol police or
> IETF gurus.  It seems to indicate only the Received: and Return-Path: lines
> are considered and are the ENVL transaction.
> 
> Lets ask this another way:
> 
> Does this proposal basically attempts to separate the transmission of a RFC
> 822 formatted message normally done at the DATA state into two separate
> state transactions?   For example:
> 
>     C: ENVL
>     S: 250 Start Mail Header input; end with <CRLF>.<CRLF>
>     C: [sends header followed by "." line]
>     S: 250 Mail Header OK. Continue with Data Command
> 
>     C: DATA
>     S: 354 Start Mail Body input; end with <CRLF>.<CRLF>
>     C: [send message body only (no header) followed by "." line]
>     S: 250 Message accepted for delivery.
> 
> or are we just adding a new state that allows a client to provide a server
> with access to "DATA" header (all or specific) information using the ENVL
> before the actual DATA transaction takes place?
> 
>     C: ENVL
>     S: 250 Start Mail (Trace??) Header input; end with <CRLF>.<CRLF>
>     C: [sends mail header (all or just some specific headers??) followed by
> "." line]
>     S: 250 Mail Header OK. Continue with Data Command
> 
>     C: DATA
>     S: 354 Start Mail Header/Body input; end with <CRLF>.<CRLF>
>     C: [sends mail header again and body followed by "." line]
>     S: 250 Message accepted for delivery.
> 
> In short,  an example for the "laymen" would be nice :-)
> 
> Anyway, I personally believe your ENVL is a good start.  It has promise.
> Off hand, without grasping yet the true intent this proposal (what does it
> solve),  what defines the ENVL "headers", etc,  I believe the answer to this
> will have a major factor on the acceptance and deployment aspect of the
> proposal.  Besides other impact considerations which I don't wish to muddy
> the water just yet without getting the answer to the above,  if this ENVL or
> another other SMTP change proposal proves to be useful,  I'll be among the
> first to implement such a new SMTP feature like this in our SMTP product   I
> will be interested on providing an immediate  "proof of concept" wide test
> bed of customers.

-- 
William Leibzon
Elan Networks
[email-masked]

From: [email-masked]
To: <[email-masked]>
Date: Fri, 23 Jan 2004 20:23:31 -0500
Sender: [email-masked]

Message-ID: <004101c3e218$87970620$6401a8c0@FAMILY>
From: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
References: <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Fri, 23 Jan 2004 20:22:26 -0500
Organization: Santronics Software, Inc
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_003E_01C3E1EE.9E18D740"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165

This is a multi-part message in MIME format.

------=_NextPart_000_003E_01C3E1EE.9E18D740
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I agree Nathaniel.  Any change to SMTP needs to be significant in what =
it addresses and if done, the opportunity should be use to address ALL =
major contemporary mail industry issues to minimize future change or =
extension requirements.

In addition, I know this may not be a popular view,  I don't believe any =
change (or extension) would have significant impact unless the =
functional specifications make it possible NOT to be BACKWARD compatible =
at the system admin policy level. In other words, allow the SysAdmin to =
decide the level of compliance. In this vain, I believe that future SMTP =
change proposals should begin with a design review on the concept of =
"client/server protocol compliant level" identification.  SMTP should be =
given a "IETF VERSION" stamp, much like HTTP 1.0/2.0, etc. and it should =
be part of the protocol handshake (or possibly DNS MX lookup) that =
clients and servers can use to determine level of compliancy.  While =
ELHO can be used to identify extensions,  deployment is minimized by not =
having an enforcement statement in the functional specifications.  =
Future SMTP operations require stronger policy base operations so that =
if a COMPANY does not want to allow for example, non-ENVL compliant =
clients, it should be allowed because the functional specification (RFC) =
allows developers to make it possible.

--=20
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

  ----- Original Message -----=20
  From: Nathaniel Borenstein=20
  To: John C Klensin=20
  Cc: [email-masked]=20
  Sent: Friday, January 23, 2004 5:52 PM
  Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

  John -- It's a pity to say this about such a nicely-written draft, but =
I fear this proposal comes remarkably close to maximizing the =
cost/benefit ratio. When you consider distributed costs of testing and =
deployment, I would bet that implementing this proposal would cost at =
*least* 50% of what it would cost to deploy a far more radical set of =
changes to the email infrastructure.=20

------=_NextPart_000_003E_01C3E1EE.9E18D740
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2800.1226" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3D"Courier New" size=3D2>I agree =
Nathaniel.&nbsp;&nbsp;Any change to=20
SMTP needs to be significant in what it addresses and if done, the =
opportunity=20
should be use to address ALL major contemporary&nbsp;mail industry =
issues to=20
minimize future change or extension requirements.</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Courier New" size=3D2>In addition, I know this may =
not be a=20
popular view,&nbsp; I don't believe any change (or extension) would have =

significant impact unless the functional specifications make it possible =
NOT to=20
be BACKWARD compatible at the system admin policy level.&nbsp;In other =
words,=20
allow the SysAdmin to decide the level of compliance.&nbsp;In this vain, =
I=20
believe that future SMTP change proposals should begin with a design =
review on=20
the concept of "client/server protocol&nbsp;compliant level"=20
identification.&nbsp; SMTP should be given a "IETF VERSION" =
stamp,&nbsp;much=20
like HTTP 1.0/2.0,&nbsp;etc.&nbsp;and it should be part of the protocol=20
handshake (or possibly DNS MX lookup) that clients and servers can use =
to=20
determine level of compliancy.&nbsp; While ELHO can be used to identify=20
extensions,&nbsp; deployment is minimized by not having an enforcement =
statement=20
in the functional specifications.&nbsp; Future SMTP operations require =
stronger=20
policy base operations so that if a COMPANY does not want to allow for =
example,=20
non-ENVL compliant clients, it should be allowed because =
the&nbsp;functional=20
specification&nbsp;(RFC) allows developers to make it =
possible.</FONT></DIV>
<DIV><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Courier New" size=3D2>-- <BR>Hector Santos, =
Santronics Software,=20
Inc.<BR><A=20
href=3D"http://www.santronics.com">http://www.santronics.com</A></FONT></=
DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT face=3D"Courier New" size=3D2></FONT>&nbsp;</DIV>
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
  <DIV style=3D"FONT: 10pt arial">----- Original Message ----- </DIV>
  <DIV=20
  style=3D"BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: =
black"><B>From:</B>=20
  <A title=[email-masked] =
href=3D"mailto:[email-masked]">Nathaniel=20
  Borenstein</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>To:</B> <A =
title=[email-masked]=20
  href=3D"mailto:[email-masked]">John C Klensin</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Cc:</B> <A =
title=[email-masked]=20
  href=3D"mailto:[email-masked]">[email-masked]</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Sent:</B> Friday, January 23, 2004 =
5:52=20
  PM</DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Subject:</B> Re: I-D=20
  ACTION:draft-klensin-email-envelope-00.txt</DIV>
  <DIV><FONT face=3D"Courier New" size=3D2></FONT><BR></DIV>John -- It's =
a pity to=20
  say this about such a nicely-written draft, but I fear this proposal =
comes=20
  remarkably close to maximizing the cost/benefit ratio. When you =
consider=20
  distributed costs of testing and deployment, I would bet that =
implementing=20
  this proposal would cost at *least* 50% of what it would cost to =
deploy a far=20
  more radical set of changes to the email infrastructure. <BR><BR>
  <BLOCKQUOTE><FONT face=3D"Courier New"=20
size=3D2></FONT>&nbsp;</BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_003E_01C3E1EE.9E18D740--

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Fri, 23 Jan 2004 20:03:56 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Hello John,

In general,  I have much to say about any SMTP proposal that suggests SMTP
change at the software level. To me, a change in software opens the door for
other possible concepts that may address a particular needs in the email
industry.   IMTV, acceptance and deployment aspects of the proposal are
important and major considerations of any SMTP change proposal.  IMTV,
that's a matter of a having a strong functional specification dictation.

With that said,  I am putting my "Software Engineering Hat" on as if I was
going to implement your ENVL proposal today to see what would be the
technical implementations issues.  Based on this technical point of view,
here are my specific comments about your draft:

What does this ENVL proposal attempt to address?  (What problem(s) does it
solve?)

Excuse me if I missed it in the draft, but I see only three (3) concepts
that addresses a "problem is solved:"

1) Proposal provides a cleaner SMTP envelope or proposal to create a cleaner
RFC 822 trace header?

2) Cleaner SMTP envelope minimizes mail relay issues.

3) Syntax Error checking.

Is this correct?

What is it about the headers today that makes it problematic?

You touch base with section 3, item 6 indicating "Syntax Error."   Syntax
Error needs to be clarified, i.e,  can (or may) this include an
implementation that attempts to perform some sort of sender/machine
validation?

What is considered the ENVL header?

The term "Trace Fields" are used.  This needs to be clarified for the
software engineer who may not be in the same league as protocol police or
IETF gurus.  It seems to indicate only the Received: and Return-Path: lines
are considered and are the ENVL transaction.

Lets ask this another way:

Does this proposal basically attempts to separate the transmission of a RFC
822 formatted message normally done at the DATA state into two separate
state transactions?   For example:

    C: ENVL
    S: 250 Start Mail Header input; end with <CRLF>.<CRLF>
    C: [sends header followed by "." line]
    S: 250 Mail Header OK. Continue with Data Command

    C: DATA
    S: 354 Start Mail Body input; end with <CRLF>.<CRLF>
    C: [send message body only (no header) followed by "." line]
    S: 250 Message accepted for delivery.

or are we just adding a new state that allows a client to provide a server
with access to "DATA" header (all or specific) information using the ENVL
before the actual DATA transaction takes place?

    C: ENVL
    S: 250 Start Mail (Trace??) Header input; end with <CRLF>.<CRLF>
    C: [sends mail header (all or just some specific headers??) followed by
"." line]
    S: 250 Mail Header OK. Continue with Data Command

    C: DATA
    S: 354 Start Mail Header/Body input; end with <CRLF>.<CRLF>
    C: [sends mail header again and body followed by "." line]
    S: 250 Message accepted for delivery.

In short,  an example for the "laymen" would be nice :-)

Anyway, I personally believe your ENVL is a good start.  It has promise.
Off hand, without grasping yet the true intent this proposal (what does it
solve),  what defines the ENVL "headers", etc,  I believe the answer to this
will have a major factor on the acceptance and deployment aspect of the
proposal.  Besides other impact considerations which I don't wish to muddy
the water just yet without getting the answer to the above,  if this ENVL or
another other SMTP change proposal proves to be useful,  I'll be among the
first to implement such a new SMTP feature like this in our SMTP product   I
will be interested on providing an immediate  "proof of concept" wide test
bed of customers.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>; <[email-masked]>
Sent: Friday, January 23, 2004 4:53 PM
Subject: FWD: I-D ACTION:draft-klensin-email-envelope-00.txt

> The draft posted/described in the attached is a bizarre idea,
> partially to see if it is possible to consider a radical
> solution to an increasingly troublesome problem, and partially
> to see if the supportive comments about "Email NG" in
> Minneapolis were really serious.
>
> I am not at all convinced that it is a _good_ idea, only that,
> if we are talking about radical changes to the mail
> infrastructure to support various extended services, this is the
> sort of "clean up the warts that get in the way" option we might
> want to consider.
>
> And even if it were a good idea, some of the details are
> probably not right -- if this looks like it received a day's
> thought, you would probably be guessing much too high.
>
> Discussion should probably go to the SMTP list; IMAA is copied
> only because this could interact a bit with some of the "UTF-8
> header" discussions.
>
>    john
>

Date: Fri, 23 Jan 2004 19:40:29 -0500
From: John C Klensin <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
cc: [email-masked]
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

Nathaniel,

We may be less in disagreement than you assume.  Briefly, I 
think it is useful to write some things up separately, because, 
that way, it is easier to think about whether the pieces are 
right or there are better ways to do them.   And I am not 
punting on internationalization at all -- it is my main concern 
here and I just want to see if we can build a healthy 
environment for it, one in which, as I've said a few times, we 
move aggressively toward an internationalized/multilingual 
Internet rather than continuing with a predominantly Roman 
character/ English network with various kludges, patches, and 
warts to permit other scripts and languages to sort of work. 
So, fwiw...

I would be happy, indeed delighted, to be wrong, but I see just 
about no hope for designing and deploying a completely new email 
structure.  More on that in draft-klensin-emailaddr-02.txt, 
which I hope I can get wrapped up and posted on Monday or 
Tuesday (it is a fairly major revision of the ideas in -01).

The nature of the internationalization problem, as I see it, is 
precisely something that will need a number of interdependent 
changes.  To try to make them independently, or to work around 
some of them, or to try for small incremental chunks, is likely 
to get us into the land of "nasty kludges forever".  I don't see 
that as flying, and I don't see it as being successful.  On the 
other hand, if we look at the whole picture (and if I'm right), 
when we get through, we have a rather different-looking email 
environment, but one that is built on the same general framework 
(avoiding hitting the wall of deploying something completely 
different).  I see that picture as including:

	* UTF-8 addressing in both the local and domain parts
	(IDNA belongs in the application-DNS interface, not in
	the user-application interface).
	
	* handling of trace information in a way that doesn't
	screw up message bodies.
	
	* probably a fundamental rethinking of the trace
	information: While the draft hints at that, it isn't a
	can of worms I wanted to open in it.  But Received is,
	at best, badly outdated.  E.g., (i) more structure may
	be in order, (ii) it is a bad sign when we have to hide
	important information in comments, (iii) Via/With/ID
	need rethinking, (iv) it might make sense to sign the
	things or make assertions about authenticity or
	authorization, (v) the whole environment may need to be
	reconsidered in the light of common practices involving
	internal and external mail hosts,... and so on.  And, of
	course, the reason I wanted to call that command
	"envelope" and not "trace" is that, if an MTA is going
	to make an assertion that the message being transported
	is somehow virtuous and in a state of grace, maybe that
	shouldn't be done by tampering with the headers either.
	
	* UTF-8 (or equivalent) headers
	
	* probably a requirement for 8BitMIME, finally
	permitting this "next generation" environment to walk
	away from the 7bit environment.

So my goal is to clean up our most problematic technical warts, 
move forward aggressively with true internationalization, and to 
provide hooks for better originator-based, authenticatable 
assertion-based, permission-based, and/or authorization-based 
control of undesired message flows (yes, in this context, I 
consider spam to be a subset of a more general problem).   And 
that is, I think, pretty close to your list of a "nice trio of 
goals".

That said, my concerns about deployment issues and a good deal 
--perhaps too much-- experience dealing with email problems 
leads me to be very hesitant to take two (perhaps obvious) extra 
steps that I think you are implying in passing:

(1) Going to true binary transport, rather than continuing with 
a line-oriented arrangement like 8BitMIME.  We pretty much know 
how to do it; I'm just concerned about the interoperability and 
debugging nightmares.

(2) Pulling all of the headers out, so that "the envelope" 
becomes:

	Outer envelope:  Handshake and addressing information
	
	Middle envelope: Transport tracing, validation, and
	similar information.
	
	Inner envelope: More or less the semantic content of the
	2822-defined headers that are not part of MIME

That way, the message payload more or less starts with a content 
type and goes from there... or one could try to abstract MIME 
structuring into an envelope layer too.  I could be wrong, but I 
just don't think the complexity and transition problems -- 
especially for gateways-- would be worth the costs.  But, if 
someone feels differently, I look forward to seeing the drafts.

      john

--On Friday, 23 January, 2004 17:52 -0500 Nathaniel Borenstein 
<[email-masked]> wrote:

> John -- It's a pity to say this about such a nicely-written
> draft, but I fear this proposal comes remarkably close to
> maximizing the cost/benefit ratio.  When you consider
> distributed costs of testing and deployment, I would bet that
> implementing this proposal would cost at *least* 50% of what
> it would cost to deploy a far more radical set of changes to
> the email infrastructure.
>
> What we were discussing in Minneapolis was a
> once-in-a-generation scale of protocol reform.  I think we
> should think big, because there is a large fixed cost to
> pushing any structural change through the entire Internet
> community.  I'm particularly concerned that you seem to be
> punting on internationalization -- I can't see putting much
> energy into *any* major reform that doesn't address the most
> widely-perceived failing of the current system.
>
> At root, what I don't buy is the notion that, in this case, we
> can take an incremental approach to radical change.  The
> Wright Brothers didn't learn to fly by practicing the high
> jump, and I don't think we're going to clean up email by
> creating a new architectural entity (the extended envelope)
> that only works in ASCII.
>
> Now, if the extended envelope could be done completely in
> UTF-8, with a coevolutionary model for son-of-822 header
> fields and binary body transport, now *that* would be a more
> compelling story.  Delivery tracing, internationalization, and
> binary transport would be a nice trio of goals for a
> next-generation email system.  I can certainly think of a few
> others, though.  -- Nathaniel
>
> On Friday, January 23, 2004, at 04:53  PM, John C Klensin
> wrote:
>
>> The draft posted/described in the attached is a bizarre idea,
>> partially to see if it is possible to consider a radical
>> solution to  an increasingly troublesome problem, and
>> partially to see if the  supportive comments about "Email NG"
>> in Minneapolis were really  serious.
>>
>> I am not at all convinced that it is a _good_ idea, only
>> that, if we  are talking about radical changes to the mail
>> infrastructure to  support various extended services, this is
>> the sort of "clean up the  warts that get in the way" option
>> we might want to consider.
>>
>> And even if it were a good idea, some of the details are
>> probably not  right -- if this looks like it received a day's
>> thought, you would  probably be guessing much too high.
>>
>> Discussion should probably go to the SMTP list; IMAA is
>> copied only  because this could interact a bit with some of
>> the "UTF-8 header"  discussions.
>>
>>   john
>>
>> From: [email-masked]
>> Date: Fri Jan 23, 2004  3:59:11  PM America/Detroit
>> To: IETF-Announce: ;
>> Subject: I-D ACTION:draft-klensin-email-envelope-00.txt
>> Reply-To: [email-masked]
>>
>>
>> A New Internet-Draft is available from the on-line
>> Internet-Drafts  directories.
>>
>>
>> 	Title		: A Cleaner SMTP Envelope for Internet Mail
>> 	Author(s)	: J. Klensin
>> 	Filename	: draft-klensin-email-envelope-00.txt
>> 	Pages		: 0
>> 	Date		: 2004-1-23
>> 	
>> During the last few years, a number of proposals for
>> extensions or improvements to email have run into trouble
>> with a side-effect of mail relaying.  In the current Internet
>> Mail model,  every SMTP server is required to break strict
>> layering by inserting  one or more additional 'trace' headers
>> into the message headers which  are actually part of the SMTP
>> payload.  Since the headers are altered in transit,
>> header-signing is not an available option, various anti-spam
>> and internationalization strategies are infeasible or much
>> more complex, and so on.  This document proposes to change the
>> Internet mail model to place the in-transit trace information
>> in the envelope, removing the requirement that relaying
>> systems modify the message payload.
>>
>> A URL for this Internet-Draft is:
>> http://www.ietf.org/internet-drafts/draft-klensin-email-envel
>> ope-00.txt
>>
>> To remove yourself from the IETF Announcement list, send a
>> message to ietf-announce-request with the word unsubscribe in
>> the body of the  message.
>>
>> Internet-Drafts are also available by anonymous FTP. Login
>> with the  username
>> "anonymous" and a password of your e-mail address. After
>> logging in, type "cd internet-drafts" and then
>> 	"get draft-klensin-email-envelope-00.txt".
>>
>> A list of Internet-Drafts directories can be found in
>> http://www.ietf.org/shadow.html
>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>
>>
>> Internet-Drafts can also be obtained by e-mail.
>>
>> Send a message to:
>> 	[email-masked].
>> In the body type:
>> 	"FILE /internet-drafts/draft-klensin-email-envelope-00.txt".
>> 	
>> NOTE:	The mail server at ietf.org can return the document in
>> 	MIME-encoded form by using the "mpack" utility.  To use this
>> 	feature, insert the command "ENCODING mime" before the "FILE"
>> 	command.  To decode the response(s), you will need "munpack"
>> 	or a MIME-compliant mail reader.  Different MIME-compliant
>> 	mail readers exhibit different behavior, especially when
>> 	dealing with "multipart" MIME messages (i.e. documents which
>> 	have been split up into multiple messages), so check your
>> 	local documentation on how to manipulate these messages.
>> 		
>> 		
>> Below is the data which will enable a MIME compliant mail
>> reader implementation to automatically retrieve the ASCII
>> version of the Internet-Draft.
>> Content-Type: text/plain
>> Content-ID:	<[email-masked]>
>>
>>

Date: Fri, 23 Jan 2004 17:52:55 -0500
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt
Cc: [email-masked]
To: John C Klensin <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

--Apple-Mail-2--45588940
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed

John -- It's a pity to say this about such a nicely-written draft, but 
I fear this proposal comes remarkably close to maximizing the 
cost/benefit ratio.  When you consider distributed costs of testing and 
deployment, I would bet that implementing this proposal would cost at 
*least* 50% of what it would cost to deploy a far more radical set of 
changes to the email infrastructure.

What we were discussing in Minneapolis was a once-in-a-generation scale 
of protocol reform.  I think we should think big, because there is a 
large fixed cost to pushing any structural change through the entire 
Internet community.  I'm particularly concerned that you seem to be 
punting on internationalization -- I can't see putting much energy into 
*any* major reform that doesn't address the most widely-perceived 
failing of the current system.

At root, what I don't buy is the notion that, in this case, we can take 
an incremental approach to radical change.  The Wright Brothers didn't 
learn to fly by practicing the high jump, and I don't think we're going 
to clean up email by creating a new architectural entity (the extended 
envelope) that only works in ASCII.

Now, if the extended envelope could be done completely in UTF-8, with a 
coevolutionary model for son-of-822 header fields and binary body 
transport, now *that* would be a more compelling story.  Delivery 
tracing, internationalization, and binary transport would be a nice 
trio of goals for a next-generation email system.  I can certainly 
think of a few others, though.  -- Nathaniel

On Friday, January 23, 2004, at 04:53  PM, John C Klensin wrote:

> The draft posted/described in the attached is a bizarre idea, 
> partially to see if it is possible to consider a radical solution to 
> an increasingly troublesome problem, and partially to see if the 
> supportive comments about "Email NG" in Minneapolis were really 
> serious.
>
> I am not at all convinced that it is a _good_ idea, only that, if we 
> are talking about radical changes to the mail infrastructure to 
> support various extended services, this is the sort of "clean up the 
> warts that get in the way" option we might want to consider.
>
> And even if it were a good idea, some of the details are probably not 
> right -- if this looks like it received a day's thought, you would 
> probably be guessing much too high.
>
> Discussion should probably go to the SMTP list; IMAA is copied only 
> because this could interact a bit with some of the "UTF-8 header" 
> discussions.
>
>   john
>
> From: [email-masked]
> Date: Fri Jan 23, 2004  3:59:11  PM America/Detroit
> To: IETF-Announce: ;
> Subject: I-D ACTION:draft-klensin-email-envelope-00.txt
> Reply-To: [email-masked]
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts 
> directories.
>
>
> 	Title		: A Cleaner SMTP Envelope for Internet Mail
> 	Author(s)	: J. Klensin
> 	Filename	: draft-klensin-email-envelope-00.txt
> 	Pages		: 0
> 	Date		: 2004-1-23
> 	
> During the last few years, a number of proposals for
> extensions or improvements to email have run into trouble with a
> side-effect of mail relaying.  In the current Internet Mail model, 
> every SMTP server is required to break strict layering by inserting 
> one or more additional 'trace' headers into the message headers which 
> are actually part of the SMTP payload.  Since the headers are
> altered in transit, header-signing is not an available option, various
> anti-spam and internationalization strategies are infeasible or much
> more complex, and so on.  This document proposes to change the
> Internet mail model to place the in-transit trace information in the
> envelope, removing the requirement that relaying systems modify the
> message payload.
>
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-klensin-email-envelope-00.txt
>
> To remove yourself from the IETF Announcement list, send a message to
> ietf-announce-request with the word unsubscribe in the body of the 
> message.
>
> Internet-Drafts are also available by anonymous FTP. Login with the 
> username
> "anonymous" and a password of your e-mail address. After logging in,
> type "cd internet-drafts" and then
> 	"get draft-klensin-email-envelope-00.txt".
>
> A list of Internet-Drafts directories can be found in
> http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
> Internet-Drafts can also be obtained by e-mail.
>
> Send a message to:
> 	[email-masked].
> In the body type:
> 	"FILE /internet-drafts/draft-klensin-email-envelope-00.txt".
> 	
> NOTE:	The mail server at ietf.org can return the document in
> 	MIME-encoded form by using the "mpack" utility.  To use this
> 	feature, insert the command "ENCODING mime" before the "FILE"
> 	command.  To decode the response(s), you will need "munpack" or
> 	a MIME-compliant mail reader.  Different MIME-compliant mail readers
> 	exhibit different behavior, especially when dealing with
> 	"multipart" MIME messages (i.e. documents which have been split
> 	up into multiple messages), so check your local documentation on
> 	how to manipulate these messages.
> 		
> 		
> Below is the data which will enable a MIME compliant mail reader
> implementation to automatically retrieve the ASCII version of the
> Internet-Draft.
> Content-Type: text/plain
> Content-ID:	<[email-masked]>
>
>

--Apple-Mail-2--45588940
Content-Transfer-Encoding: 7bit
Content-Type: text/enriched;
	charset=US-ASCII

John -- It's a pity to say this about such a nicely-written draft, but
I fear this proposal comes remarkably close to maximizing the
cost/benefit ratio.  When you consider distributed costs of testing
and deployment, I would bet that implementing this proposal would cost
at *least* 50% of what it would cost to deploy a far more radical set
of changes to the email infrastructure.  

What we were discussing in Minneapolis was a once-in-a-generation
scale of protocol reform.  I think we should think big, because there
is a large fixed cost to pushing any structural change through the
entire Internet community.  I'm particularly concerned that you seem
to be punting on internationalization -- I can't see putting much
energy into *any* major reform that doesn't address the most
widely-perceived failing of the current system.

At root, what I don't buy is the notion that, in this case, we can
take an incremental approach to radical change.  The Wright Brothers
didn't learn to fly by practicing the high jump, and I don't think
we're going to clean up email by creating a new architectural entity
(the extended envelope) that only works in ASCII.

Now, if the extended envelope could be done completely in UTF-8, with
a coevolutionary model for son-of-822 header fields and binary body
transport, now *that* would be a more compelling story.  Delivery
tracing, internationalization, and binary transport would be a nice
trio of goals for a next-generation email system.  I can certainly
think of a few others, though.  -- Nathaniel

On Friday, January 23, 2004, at 04:53  PM, John C Klensin wrote:

<excerpt>The draft posted/described in the attached is a bizarre idea,
partially to see if it is possible to consider a radical solution to
an increasingly troublesome problem, and partially to see if the
supportive comments about "Email NG" in Minneapolis were really
serious.

I am not at all convinced that it is a _good_ idea, only that, if we
are talking about radical changes to the mail infrastructure to
support various extended services, this is the sort of "clean up the
warts that get in the way" option we might want to consider.

And even if it were a good idea, some of the details are probably not
right -- if this looks like it received a day's thought, you would
probably be guessing much too high.

Discussion should probably go to the SMTP list; IMAA is copied only
because this could interact a bit with some of the "UTF-8 header"
discussions.

  john

<bold>

From: </bold>[email-masked]

<bold>Date: </bold>Fri Jan 23, 2004  3:59:11  PM America/Detroit

<bold>To: </bold>IETF-Announce: ;

<bold>Subject: </bold>I-D ACTION:draft-klensin-email-envelope-00.txt

<bold>Reply-To: </bold>[email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title		: A Cleaner SMTP Envelope for Internet Mail

	Author(s)	: J. Klensin

	Filename	: draft-klensin-email-envelope-00.txt

	Pages		: 0

	Date		: 2004-1-23

During the last few years, a number of proposals for

extensions or improvements to email have run into trouble with a

side-effect of mail relaying.  In the current Internet Mail model,
every SMTP server is required to break strict layering by inserting
one or more additional 'trace' headers into the message headers which
are actually part of the SMTP payload.  Since the headers are

altered in transit, header-signing is not an available option, various

anti-spam and internationalization strategies are infeasible or much

more complex, and so on.  This document proposes to change the

Internet mail model to place the in-transit trace information in the

envelope, removing the requirement that relaying systems modify the

message payload.

A URL for this Internet-Draft is:

http://www.ietf.org/internet-drafts/draft-klensin-email-envelope-00.txt

To remove yourself from the IETF Announcement list, send a message to 

ietf-announce-request with the word unsubscribe in the body of the
message.

Internet-Drafts are also available by anonymous FTP. Login with the
username

"anonymous" and a password of your e-mail address. After logging in,

type "cd internet-drafts" and then

	"get draft-klensin-email-envelope-00.txt".

A list of Internet-Drafts directories can be found in

http://www.ietf.org/shadow.html 

or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:

	[email-masked].

In the body type:

	"FILE /internet-drafts/draft-klensin-email-envelope-00.txt".

NOTE:	The mail server at ietf.org can return the document in

	MIME-encoded form by using the "mpack" utility.  To use this

	feature, insert the command "ENCODING mime" before the "FILE"

	command.  To decode the response(s), you will need "munpack" or

	a MIME-compliant mail reader.  Different MIME-compliant mail readers

	exhibit different behavior, especially when dealing with

	"multipart" MIME messages (i.e. documents which have been split

	up into multiple messages), so check your local documentation on

	how to manipulate these messages.

		

Below is the data which will enable a MIME compliant mail reader

implementation to automatically retrieve the ASCII version of the

Internet-Draft.

<fixed>Content-Type: text/plain

Content-ID:	<<[email-masked]>

</fixed></excerpt>
--Apple-Mail-2--45588940--

Date: Fri, 23 Jan 2004 16:53:33 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], [email-masked]
Subject: FWD: I-D ACTION:draft-klensin-email-envelope-00.txt
Sender: [email-masked]

--==========205C0896BCAF102B84B0==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

The draft posted/described in the attached is a bizarre idea, 
partially to see if it is possible to consider a radical 
solution to an increasingly troublesome problem, and partially 
to see if the supportive comments about "Email NG" in 
Minneapolis were really serious.

I am not at all convinced that it is a _good_ idea, only that, 
if we are talking about radical changes to the mail 
infrastructure to support various extended services, this is the 
sort of "clean up the warts that get in the way" option we might 
want to consider.

And even if it were a good idea, some of the details are 
probably not right -- if this looks like it received a day's 
thought, you would probably be guessing much too high.

Discussion should probably go to the SMTP list; IMAA is copied 
only because this could interact a bit with some of the "UTF-8 
header" discussions.

   john

--==========205C0896BCAF102B84B0==========
Content-Type: message/rfc822;
 name="I-D ACTION:draft-klensin-email-envelope-00.txt"

Delivery-date: Fri, 23 Jan 2004 16:17:32 -0500
To: IETF-Announce: ;
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-klensin-email-envelope-00.txt
Date: Fri, 23 Jan 2004 15:59:11 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: A Cleaner SMTP Envelope for Internet Mail
	Author(s)	: J. Klensin
	Filename	: draft-klensin-email-envelope-00.txt
	Pages		: 0
	Date		: 2004-1-23
	
During the last few years, a number of proposals for
extensions or improvements to email have run into trouble with a
side-effect of mail relaying.  In the current Internet Mail model, every SMTP server is required to break strict layering by inserting one or more additional 'trace' headers into the message headers which are actually part of the SMTP payload.  Since the headers are
altered in transit, header-signing is not an available option, various
anti-spam and internationalization strategies are infeasible or much
more complex, and so on.  This document proposes to change the
Internet mail model to place the in-transit trace information in the
envelope, removing the requirement that relaying systems modify the
message payload.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-klensin-email-envelope-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-klensin-email-envelope-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-klensin-email-envelope-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-klensin-email-envelope-00.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-klensin-email-envelope-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

--==========205C0896BCAF102B84B0==========--

Date: Thu, 22 Jan 2004 16:12:49 -0800 (PST)
From: [email-masked]
Subject: Re: Multi-line Greetings - where to put official greeting line (Top or Bottom?)
To: "B. Johannessen" <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Sender: [email-masked]

> I did however discover one thing. I was sending the initial greeting
> via 2 write sys-calls. Think of it like this:

> write(fd, "220 h.db.org ESMTP ready", 24);
> ...
> ...
> write(fd, "\r\n", 2);

> This confused the "Mail Test" to the point where it claimed not to
> have received a valid response. Now I'm no expert, but I can't remember
> having read anything that mandates that the whole response be sent in
> one TCP packet.

That's because there is no such mandate. And even if there were such a mandate
it could not be enforced from an application since how a TCP stack decides to
map I/O operations into packets on the wire can and does vary. (Some
implementations allow some degree of control over this but others do not.)

> With the exception of spam sending malware it doesn't
> seem to be a problem for anyone else sending me mail.

Right, because implementations are supposed to be tolerant in this regard.

> Can someone confirm that it is acceptable to send SMTP responses divided
> over several TCP packets?

It is perfectly acceptable, although the converse is more commonly
seen, that is, multiple responses being sent in a single packet. (This
has also been known to cause problems with incompliant implementations.)

				Ned

Subject: Re: Multi-line Greetings - where to put official greeting line (Top or Bottom?)
From: "B. Johannessen" <[email-masked]>
To: IETF-SMTP <[email-masked]>
Organization: http://db.org/
Date: Thu, 22 Jan 2004 23:49:57 +0000
Sender: [email-masked]

On Tue, 2004-01-20 at 16:59, Hector Santos wrote:
> We got a question from a customer saying DNSREPORT.COM
> (http://www.dnsreport.com)  is reporting a warning abort a host name
> mismatch in the greeting.  DNSREPORT.COM is a popular SMTP compliancy
> testing site.

I just tested it (dnsreport.com "Mail Test") against a qmail-smtpd
replacement I'm working on. The "test" consists of the following
protocol exchange:

>>>> 220 h.db.org ESMTP ready
<<<< HELO test.DNSreport.com
>>>> 250 h.db.org
<<<< MAIL FROM:<[email-masked]>
>>>> 250 2.1.0 sender <[email-masked]> ok
<<<< RCPT TO:<[email-masked]>
>>>> 250 2.1.5 recipient <[email-masked]> ok
<<<< QUIT
>>>> 221 2.0.0 h.db.org goodbye

I did however discover one thing. I was sending the initial greeting
via 2 write sys-calls. Think of it like this:

write(fd, "220 h.db.org ESMTP ready", 24);
...
...
write(fd, "\r\n", 2);

This confused the "Mail Test" to the point where it claimed not to
have received a valid response. Now I'm no expert, but I can't remember
having read anything that mandates that the whole response be sent in
one TCP packet. With the exception of spam sending malware it doesn't
seem to be a problem for anyone else sending me mail.

Can someone confirm that it is acceptable to send SMTP responses divided
over several TCP packets?

	Bob

BTW: As i mentioned, spam zombies seems to have the same problem, so
maybe this could be the next anti-spam trick (for about a week or so,
before the spammers catch up).

-- 
-=[ B. Johannessen | [email-masked] -=- http://db.org/ | +4797152009 ]=-
-=[ Mail & Spam - News, Drafts & Standards - http://db.org/blog/ ]=-
-=[ On the Origin Of Spam: Spam Statistics - http://db.org/spam/ ]=-

Date: Wed, 21 Jan 2004 23:52:31 +0100
From: [email-masked]
To: IETF-SMTP <[email-masked]>
Cc: Hector Santos <[email-masked]>
Subject: Re: Multi-line Greetings - where to put official greeting line (Top or Bottom?)
Mail-Followup-To: [email-masked], IETF-SMTP <[email-masked]>, Hector Santos <[email-masked]>
Sender: [email-masked]

Hector,

I'm not the guru and actually might be wrong, but I believe the first
line should contain the hostname, just as you did it. This is also how,
for example, sendmail does it when using multiline greeting messages.
RFC2821 does not explicitly mention this and the info in the RFC seems,
at least to me, a bit confusing:

4.1.1.1:
  [...]
      ehlo-ok-rsp  =    ( "250"    domain [ SP ehlo-greet ] CRLF )
                   / (    "250-"   domain [ SP ehlo-greet ] CRLF
                       *( "250-"   ehlo-line                CRLF )
                          "250"    SP ehlo-line             CRLF  )
  [...]

For the 250 reply code, the data in the first line stays the same, no
matter whether it is single line or multiline. Somehow it would also be
counterintuitive to move the first line to the end in multilines.

But oops, the SP right after the 250 code is missing in the single line
version. A documentation bug?

4.2
  [...]
  In particular, the 220, 221, 251, 421, and 551 reply codes
  are associated with message text that must be parsed and interpreted
  by machines.
  [...]
  Formally, a reply is defined to be the sequence: a
  three-digit code, <SP>, one line of text, and <CRLF>, or a multiline
  reply (as defined in section 4.2.1)
  [...]
  Only the EHLO, EXPN, and HELP
  commands are expected to result in multiline replies in normal
  circumstances, however, multiline replies are allowed for any
  command.
  [...]

Should a multiline greeting be allowed as well, even it is not the
reply for a command, but rather the reply of the connection initiation?
I believe yes.

4.2
  [...]
      Greeting = "220 " Domain [ SP text ] CRLF
      Reply-line = Reply-code [ SP text ] CRLF
  [...]

The ABNF for the greeting, does include a space right after
the reply code and there is no multi-line variant given. Possibly
this ABNF should be adjusted to
      Greeting =   (    "220"   SP domain [ SP text ] CRLF   )
                 / (    "220-"     domain [ SP text ] CRLF
                     *( "220-"                 text   CRLF )
                        "220"   SP           [ text ] CRLF   )

4.2.1
  [...]
  The reply text may be longer than a single line; in these cases the
  complete text must be marked so the SMTP client knows when it can
  stop reading the reply.  This requires a special format to indicate a
  multiple line reply.

  The format for multiline replies requires that every line, except the
  last, begin with the reply code, followed immediately by a hyphen,
  "-" (also known as minus), followed by text.  The last line will
  begin with the reply code, followed immediately by <SP>, optionally
  some text, and <CRLF>.  As noted above, servers SHOULD send the <SP>
  if subsequent text is not sent, but clients MUST be prepared for it
  to be omitted.
  [...]
  In many cases the SMTP client then simply needs to search for a line
  beginning with the reply code followed by <SP> or <CRLF> and ignore
  all preceding lines.  In a few cases, there is important data for the
  client in the reply "text".  The client will be able to identify
  these cases from the current context.

The last paragraph probably only should be applicable if the client does
not need/want to interpret any of the data in the reply but simply wants
to skip over it. It is likely the implementation you encountered is
either skipping to the last line of the greeting or is looking for a 220
reply code which is followed immediately by a space. Would be
interesting to know what that implementation does when a 554 is given
for the greeting, rather than a 220.

All in all I would think the domain has to be looked for in the first
line of the greeting, even if it is a multiline greeting.

Cheers,

Kurt

On Tue, Jan 20, 2004 at 11:59:36AM -0500, Hector Santos wrote:
> 
> 220-winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
> 220-************** WARNING:  FOR AUTHORIZED USE ONLY! **********************
> 220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY COMPUTERS    *
> 220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE UNSOLICITED *
> 220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT ACCESS *
> 220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.                      *
> 220 ************************************************************************
> 
> [...]
>         mail.winserver.com claims to be host **************.
> 
> It seems DNSREPORT.COM is only looking at the last line to get the
> information it expects.
> 
> I tested this theory by manually adding a simulated SMTP greeting line that
> looks like the top line to the bottom of the system policy:
> 
> 220-winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
> 220-************** WARNING:  FOR AUTHORIZED USE ONLY! **********************
> 220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY COMPUTERS    *
> 220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE UNSOLICITED *
> 220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT ACCESS *
> 220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.                      *
> 220-************************************************************************
> 220 winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
> 
> and this time it got pass this particular SMTP greeting/hostname test.
> 
> The question I have is what is the correct logic?
> 
> Where should the offical SMTP greeting line be in a multi-line greeting
> display?  Top or Bottom?  Both maybe for compatibility?   Unless I missed
> it,  RFC 2821 does not touch on this particular subject item.

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Multi-line Greetings - where to put official greeting line (Top or Bottom?)
Date: Tue, 20 Jan 2004 11:59:36 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

We recently added an option for the display of a system policy file at the
SMTP greeting.

The logic is to first display the standard SMTP greeting line:

       220-hostdomain [product info] Server Ready

followed by admin's system policy file (if found) as extra 220 response
lines.   The last line, of course, does not contain the dash. For example:

220-winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
220-************** WARNING:  FOR AUTHORIZED USE ONLY! **********************
220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY COMPUTERS    *
220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE UNSOLICITED *
220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT ACCESS *
220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.                      *
220 ************************************************************************

We got a question from a customer saying DNSREPORT.COM
(http://www.dnsreport.com)  is reporting a warning abort a host name
mismatch in the greeting.  DNSREPORT.COM is a popular SMTP compliancy
testing site.

Checking it out for myself against my system, it reports:

        WARNING: One or more of your mailservers may be claiming to be a
host other than what
        it really is (the SMTP greeting should be a 3-digit code, followed
by a space or a dash, then the
        host name). This probably won't cause any harm, but may be a
technical violation of RFC821 4.3.

        mail.winserver.com claims to be host **************.

It seems DNSREPORT.COM is only looking at the last line to get the
information it expects.

I tested this theory by manually adding a simulated SMTP greeting line that
looks like the top line to the bottom of the system policy:

220-winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
220-************** WARNING:  FOR AUTHORIZED USE ONLY! **********************
220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY COMPUTERS    *
220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE UNSOLICITED *
220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT ACCESS *
220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY.                      *
220-************************************************************************
220 winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready

and this time it got pass this particular SMTP greeting/hostname test.

The question I have is what is the correct logic?

Where should the offical SMTP greeting line be in a multi-line greeting
display?  Top or Bottom?  Both maybe for compatibility?   Unless I missed
it,  RFC 2821 does not touch on this particular subject item.

Thanks in advance

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
From: [email-masked]
Date: Fri, 16 Jan 2004 13:19:53 -0500
Sender: [email-masked]

--==_Exmh_-996877281P
Content-Type: text/plain; charset=us-ascii

On Fri, 16 Jan 2004 13:01:45 EST, Hector Santos said:
> HELO: 
>           allows for quick sender/machine validation

You seem to have this hangup on needing to use the HELO/EHLO for validation,
when (a) it's *KNOWN* to be incorrect/broken in some cases and (b) you already
have *THE ACTUAL IP ADDRESS*, which is more likely to give you something
usable. 

>From the message you replied to:

Received: from turing-police.cc.vt.edu (h80ad2659.async.vt.edu [128.173.38.89])
 	by above.proper.com (8.12.10/8.12.8) with ESMTP id i0GHbIib009295
	for  <[email-masked]>; Fri, 16 Jan 2004 09:37:19 -0800

Yes, my laptop has a hostname.  No, it doesn't correspond to the DNS reality
unless it's in its docking station in my office. Yes, it's on a dialup at the
moment. Yes, it gave the hostname it thought it had on the EHLO.  No, trying to
double- check what it said on the ELHO won't do you much good, that will just
point you back at the docking station in my office. Yes, trying to chase back
the actual IP address and/or the PTR for the IP address is more likely to do
you any good.

*NOW* do you see why we object to making the protocol more complicated just so
you can double check something that's NOT a good validator of ANYTHING?

--==_Exmh_-996877281P
Content-Type: application/pgp-signature

--==_Exmh_-996877281P--

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
Date: Fri, 16 Jan 2004 13:01:45 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Friday, January 16, 2004 12:37 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

On Fri, 16 Jan 2004 11:28:44 EST, Hector Santos
<[email-masked]>  said:

> So we stick an 'updates 2821' on it.  Isn't like we haven't done THAT with
a
> number of protocols in the past.

As long it has it makes it possible for enforce validation,  Ok, I guess
this ends the discussion and any talk to change 2821 in this area.

>> (Incidentally, exactly these sort of bogosities are why the spec didn't
allow rejection
>> based on EHLO in the first place.)

Oy vey, I appreciate the historically perceptive. :-)

But if your "car" is now broken,  you lost the keys, you are out of gas,
money and friends.  You now need to find another way to get where you want
to go!

Maybe, what we need to do is change the order of the state machine to define
your ultimate intention?

RCPT TO:

          refused recipient, don't bother with the rest, optimizes, improved
SMTP engine with eliminating
          of validation overhead.

HELO:

          allows for quick sender/machine validation

MAIL FROM:

         allows for return path validation.

DATA

The point is,  you are debating a time warped or legacy methodology that
simply doesn't apply any more (for the most part) and at the same time seem
to be advocating (or instilling the idea) a need or requirement for change.
When people have problem, there are other ways for contact.   The beauty of
course, is that the people we are trying to protect against, won't bother
you either way.

You can't have it both ways!

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
From: [email-masked]
Date: Fri, 16 Jan 2004 12:37:16 -0500
Sender: [email-masked]

--==_Exmh_-1040228612P
Content-Type: text/plain; charset=us-ascii

On Fri, 16 Jan 2004 11:28:44 EST, Hector Santos <[email-masked]>  said:

> For example, if the ASRG efforts finally put out the official
> recommendations (which to me, looks like its DNS lookup based, hence sender
> machine),  then this along says that it will be conflictive with the current
> HELO/EHLO functional specifications. For example,  the following RFC 2821
> will now be inconsistent with the IRTF drafts:

So we stick an 'updates 2821' on it.  Isn't like we haven't done THAT with a
number of protocols in the past.

>  A server MAY refuse to accept a message if the verification fails, however, this
>   non-acceptance of mail failure SHOULD be based on a SENDER MACHINE
>   validation technique.  The information about any verification  failure SHOULD
>   recorded for logging and tracing.

And language like this is *exactly* why the Protocol Police are very leery of
this kind of change.  As currently written, you are NOT allowed to reject based
on the contents of an EHLO - the most you can do is (optionally) warn/log it.

As written, a mail server would be allowed to reject a message, and not explain
to the sender *why* - you're missing a 'If a message is rejected, the server
MUST include an explanatory text in the 5xx reply'.

Also, you're neglecting a VERY important corner case - one of your OWN users
has egregiously misconfigured their machine, causing your mail server to not
let them send a "how do I fix this" to somebody.  You forgot to add language
saying that mail to POSTMASTER must always be allowed through.

Of course, this means that you can't reject at the EHLO, you have to wait till
the RCPT TO: stage - or possibly DATA (and then you get to find out how many
end users are running crapware MUAs that don't know that anything other than a
354 response to DATA is possible ;)

(Incidentally, exactly these sort of bogosities are why the spec didn't allow rejection
based on EHLO in the first place.)

--==_Exmh_-1040228612P
Content-Type: application/pgp-signature

--==_Exmh_-1040228612P--

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Fri, 16 Jan 2004 11:28:44 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>
Sent: Friday, January 16, 2004 9:27 AM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

> > Section  3.3 makes it possible to perform return path validation;
>
> yes, it's possible.  that doesn't mean your way of determining whether
> the reverse-path is acceptable is a good idea.

Keith,  we written directly to each other on this.

I feel it is important to focus on this aspect of the topic publically.

It also doesn't mean it (CBV) is a bad idea either.

Nonetheless, as I indicated a few times, I am not bent on any idea.  I am
very conservative too.  Our main goal was to provide a black box validation
system into each state of our SMTP engine.  That has been accomplished and
we used a CBV along with other sender-machine validation concepts to provide
an SMTP level, non-mail content related, access control system.

Rhetorically, do you have a better way today that is consistent with the
current SMTP protocol?  Immediately deployable?

If so, then lets talk about it.

If not,  forget about CBV,   lets talk about how it can be done.   That is
why I am here.   I really don't want to waste anyone's time here or my own.
Is this a group of industry developers?   If this is not the place, does
anyone have a place or where I can find a group of industry developers,  not
policy makers or administrators,  to bang our heads together and come up
with a best backward compatible solution/proposal?   If this has already
been done or if you have your own view on this, then please,  enlighten me.

Forgive me for using your original message to returning the carrot on this
topic.  It was pretty obvious your questions were more rhetorical and your
mindset and conclusion was foretold.  I understand 100% what are all the
issues involved with CBV.  I haven't heard anything that I was not aware
about.  However,  if it was not clear,  my main goal was NOT to prove that
CBV is the ultimate solution, it isn't,  but rather  to show how it was used
as a proof of concept to illustrate the MAIN issues with the functional
specifications of SMTP protocol.

The CBV is really a multi-point testing system.  One excellent discussion
that was brought up elsewhere is how it compares to a DNS Lookup proposals.

The CBV validates the email domain, the domain MX,  the domain IPs, the IP
connection and the SMTP session.

The DNS lookup methods validates only the sender machine.

The questions are:

What is more important to provide access control?  Sender machine,  Return
path or both?

In my research, you need to consider both.  Putting aside the possible
strength of each one,  the problem for developers will be how any validation
conflicts with the functional specification.

For example, if the ASRG efforts finally put out the official
recommendations (which to me, looks like its DNS lookup based, hence sender
machine),  then this along says that it will be conflictive with the current
HELO/EHLO functional specifications. For example,  the following RFC 2821
will now be inconsistent with the IRTF drafts:

4.1.4 Order of Commands

   ...

   An SMTP server MAY verify that the domain name parameter in the EHLO
   command actually corresponds to the IP address of the client.
   However, the server MUST NOT refuse to accept a message for this
   reason if the verification fails: the information about verification
   failure is for logging and tracing only.

A possible rephrase would be:

   An SMTP server MAY verify that the domain name parameter in the EHLO
   command actually corresponds to the IP address of the client or possible
using
   one or more SENDER MACHINE validation techniques [See XXXX].

   A server MAY refuse to accept a message if the verification fails,
however, this
   non-acceptance of mail failure SHOULD be based on a SENDER MACHINE
   validation technique.  The information about any verification  failure
SHOULD
   recorded for logging and tracing.

Do you see what I mean?   The same is true across the board for other
validation concepts in the functional specification, such as the return
path.  I don't think the sender-machine validation is sufficient in
controlling anonymous access abuse.  The return path validation is the next
possible "current SMTP protocol" technique.   We will need the same
rephrasing of return path validation in the specs, because some day Keith,
if its not CBV, there will be another more optimized method.

Otherwise, what will happen is that there will be deployment problems and
there will certainly be many arguments and debates who's software is
"broken."

I hope I have made myself clearer here.

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for disabled commands 
From: [email-masked]
Date: Fri, 16 Jan 2004 10:31:26 -0500
Sender: [email-masked]

--==_Exmh_-1148599045P
Content-Type: text/plain; charset=us-ascii

On Thu, 15 Jan 2004 15:20:26 EST, Hector Santos <[email-masked]>  said:

> Hmmmmm, unless I am wrong,  I was under the current/modern SMTP design
> assumption the initiating "Message Sender" must remain constant throughout
> the entire multi-host route and that the Return-Path: header is only added
> during final delivery.     This has to be true in order for the final
> delivery to stamp the proper Return Path header with the proper return path.
> 
> Am I wrong?

Yes.

It is quite possible that an intermediate system (most likely in the DMZ of a
firewalled company) will find it necessary to source-route or %-hack the
SMTP MAIL FROM such that the final value will be deliverable at
the destination.  Yes, I know 2821, section 6.1 has a MUST about where
to send the notification for source-routed addresses, but (a) it doesn't
apply to %-hacked addresses and (b) there's probably still a lot of places
that can't do it right (i.e. the interior system doesn't know how to route
a non-source-routed address).

Note that this is important for any CBV system, as you're making the
assumption that the call-back can be done in real time even though the
mail you're trying to verify came in via a store-and-forward.

> I believe the statement clearly presumes that an unambiguous return-path
> MUST (not SHOULD) be valid in order for  the delivery of  a"system failure"
> message can be attempted.

The fact that the return-path was valid to within the best-effort attempts
at the time the mail was sent does *not* imply that it MUST (in the RFC sense)
be valid later at some unspecified day.

Back in December, I sent out a mass mailing with a specially crafted MAIL FROM
so bounces went to a controlled location.  Said address is no longer valid - I disabled
it a week after I did the mailing.  Where does this fit in your 'MUST be valid'?
It's quite possible some borked system took 8 days to bounce a mail, and the
bounce mailbox evaporated before the bounce mail arrived.

> > AFAIK MS Exchange does verify the syntax of an address during SMTP
> > transmission, but _not_ the existence of the recipient address.
> 
> I believe this is a matter of implementation as it is possible to hook into
> the RCPT TO: state.

RFC2821, section 6.1:

   Some delivery failures after the message is accepted by SMTP will be
   unavoidable.  For example, it may be impossible for the receiving
   SMTP server to validate all the delivery addresses in RCPT command(s)
   due to a "soft" domain system error, because the target is a mailing
   list (see earlier discussion of RCPT), or because the server is
   acting as a relay and has no immediate access to the delivering
   system.

In other words, if you're down and your secondary MX grabs it, it probably
can't cough up a User Unknown. And then of course you have to decide how to do
CBV if a secondary MX is forwarding...

> what happens once the data is received.  If we are to believe AOL.COM's
> claim as the largest database of email users, then why is it they do dynamic
> validation of the recipient address?   Why don't they delay the process?

Very simple - if they '250 OK' the mail, they commit to taking responsibility for
the mail.  When you're rejecting a billion spams a day, the LAST thing you want
to do is say '250 OK' and then get stuck having to deal with the message.
There's a lot of sins that the spec permits before you 250 it that you can't do
once you 250 it:

   When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
   message in response to DATA), it is accepting responsibility for
   delivering or relaying the message.  It must take this responsibility
   seriously.  It MUST NOT lose the message for frivolous reasons, such
   as because the host later crashes or because of a predictable
   resource shortage.

If you look at the Sendmail code, there's a number of places where it hits
an extremely performance-painful fsync() or fdirsync() call, simply to protect
against crashes. All that extra I/O *hurts* when you're doing high-throughput
mail systems.

--==_Exmh_-1148599045P
Content-Type: application/pgp-signature

--==_Exmh_-1148599045P--

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for
Date: Fri, 16 Jan 2004 02:19:31 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "IETF-SMTP" <[email-masked]>
Sent: Thursday, January 15, 2004 11:23 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for
disabled commands

> However messages with invalid return-paths *can* have a legitimate
> purpose.

Keith,

I guess it is possible for one or more pieces of software use the same
weakness in the functional specs that spammers exploit for what is deemed
legitimate behavior.  I don't wish to mis-understand you so correct me if I
am wrong.  Are you stating this is considered normal, standard,
pseudo-standard, legacy or compliant behavior and therefore,  proposals to
that remove this possibility ought not be considered?

> An invalid return-path should have no effect at all when the message is
> able to be successfully delivered.

Hence you are saying that validity is not in question until message can not
be delivered?   Which is why I ask, when it is require to be valid?   If you
consider bounce mail delivery is also part of the functional requirement it
has a major effect on the system.

> >  Ok,  so what if a DSN was going to be sent?   A normal SMTP queuing
> > will occur.
>
> If, once you've decided to bounce a message, you want to use CBV to
> decide whether to deliver a DSN, feel free, though I don't really see
> why you'd bother.

I was referring to a normal SMTP send mail process queuing outbound bounce
mail.   The CBV is not going queue the attempt to validate a session return
path.

> Good question.  Some countermeasures are more easily circumvented than
> others, so it might make sense to deploy mechanisms that aren't easily
> circumvented.

Of course, anything that works better, is preferrable.  Nonetheless,  even
these implementations, if and when successful, it will considered  wasteful
overhead.

> Client/sender validation might well be a good idea.  But we shouldn't
> confuse that with return-path validation.

I poise the question again, What is the purpose of the return path?  When it
is suppose to be valid or not?   If you say it *may not* be valid, then that
should be written into functional specification.   If you saying that for
legacy reasons, there might be situations where an invalid return path is
used for some kludge behavior, well, I don't think we can work on this basis
anymore.  You can't have it both ways and expect auditing and tractable
solutions to be developed.

Section  3.3 makes it possible to perform return path validation;

3.3 Mail Transactions

   ...

   The first step in the procedure is the MAIL command.

      MAIL FROM:<reverse-path> [SP <mail-parameters> ] <CRLF>

    ..................................  If accepted, the SMTP server returns
   a 250 OK reply.  If the mailbox specification is not acceptable for
   some reason, the server MUST return a reply indicating whether the
   failure is permanent (i.e., will occur again if the client tries to
   send the same address again) or temporary (i.e., the address might be
   accepted if the client tries again later).

   ...............................................Despite the apparent
   scope of this requirement, there are circumstances in which the
   acceptability of the reverse-path may not be determined until one or
   more forward-paths (in RCPT commands) can be examined.  In those
   cases, the server MAY reasonably accept the reverse-path (with a 250
   reply) and then report problems after the forward-paths are received
   and examined.  Normally, failures produce 550 or 553 replies.

I guess what needs to be defined is "acceptability."

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Cc: Keith Moore <[email-masked]>, "IETF-SMTP" <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Thu, 15 Jan 2004 23:23:42 -0500
To: "Hector Santos" <[email-masked]>
Sender: [email-masked]

>> if widely adopted CBV will impair the reliability of the mail system 
>> by
>> causing delivery failures for legitimate messages in cases where
>> delivery would otherwise have been successful (i.e. if the return-path
>> field was invalid or not usable due to a configuration error or
>> temporary failure)
>
> [Side note: I think the word legitimate does not play a role here as 
> this
> can happen in general.  The process has no information on whether the
> transaction is legitimate, if it did, then we this CBV does not apply]

The (unstated) assumption is that the reliability of the mail system is 
defined in terms of how many legitimate messages were delivered vs. how 
many were sent.  "Legitimate" is basically trying to encapsulate the 
idea that it's reasonable for the mail system to deliberately fail to 
deliver certain kinds of messages that have no legitimate purpose such 
as viruses and spam (at least, messages that can be reliably identified 
as spam).

However messages with invalid return-paths *can* have a legitimate 
purpose.

> I fail to see where an invalid field would not be a problem in the 
> first
> place.

An invalid return-path should have no effect at all when the message is 
able to be successfully delivered.

>  Ok,  so what if a DSN was going to be sent?   A normal SMTP queuing 
> will occur.

If, once you've decided to bounce a message, you want to use CBV to 
decide whether to deliver a DSN, feel free, though I don't really see 
why you'd bother.

>> since CBV is easily circumvented by spammers, in the long term CBV 
>> does
>> no good - it only introduces overhead.
>
> Well,  IMO, this ideology applies to any solution, including those 
> that may
> be or not capable of being circumvented.  In other words, if a solution
> works then it inevitably becomes overhead and redundancy.
>
> So what's the point?

Good question.  Some countermeasures are more easily circumvented than 
others, so it might make sense to deploy mechanisms that aren't easily 
circumvented.   But a significant set of the spam countermeasures that 
people are deploying are pretty dubious anyway and probably shouldn't 
be deployed on a widespread basis.  You also should consider how long 
it takes to deploy the countermeasure vs. how long it's likely to last 
- for something like CBV the spammers can circumvent it before you get 
it widely deployed.  Finally, you should consider that it's much harder 
to get rid of features like CBV - even when they cause harm - than it 
is to deploy them.

Basically my concern is that poorly chosen countermeasures for spam and 
viruses are doing serious harm to the reliability of the mail system, 
so I want to discourage people from deploying them.

> I think the important points in the debate is the exploitation of the
> functional specification, not CBV.  As long as the specs remain 
> relaxed on
> client/sender validation, we are never going to solve anything and 
> those
> that do provide solutions will be accused of having "broken software."

Client/sender validation might well be a good idea.  But we shouldn't 
confuse that with return-path validation.

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Thu, 15 Jan 2004 21:33:55 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "IETF-SMTP" <[email-masked]>
Sent: Thursday, January 15, 2004 8:26 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

> I could reply to your reply in detail but that would only distract from
> the important points in this debate.

Thanks, however.....

> the points are these:
>
> if widely adopted CBV will impair the reliability of the mail system by
> causing delivery failures for legitimate messages in cases where
> delivery would otherwise have been successful (i.e. if the return-path
> field was invalid or not usable due to a configuration error or
> temporary failure)

[Side note: I think the word legitimate does not play a role here as this
can happen in general.  The process has no information on whether the
transaction is legitimate, if it did, then we this CBV does not apply]

I fail to see where an invalid field would not be a problem in the first
place.  Ok,  so what if a DSN was going to be sent?   A normal SMTP queuing
will occur.

Scenerio:

Real Bad System:    Non-CBV system will have a wasteful long term retry
history.

In the CBV, based on the implementation,   I have it so its sysop
configurable:

    ConnectFailure  Reject |  Accept  [MaximiumAllowed [5]]

Rejection codes are definable as well.

So I don't see this is as a reason to reject the concept.  It is technically
addressable.  Fortunately, like I said, most systems are compliant.  So
those that are problematic are more than likely to be those systems were are
trying to protect against.  And those with false negatives are sent 45x.
The legit system will try again.   I have not seen one report that indicated
this is a problem, and unexpectively so, because spammers are spoofing other
domains.

> since CBV is easily circumvented by spammers, in the long term CBV does
> no good - it only introduces overhead.

Well,  IMO, this ideology applies to any solution, including those that may
be or not capable of being circumvented.  In other words, if a solution
works then it inevitably becomes overhead and redundancy.

So what's the point?

I think the important points in the debate is the exploitation of the
functional specification, not CBV.  As long as the specs remain relaxed on
client/sender validation, we are never going to solve anything and those
that do provide solutions will be accused of having "broken software."

Thanks for your comments Keith.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Cc: Keith Moore <[email-masked]>, "IETF-SMTP" <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Thu, 15 Jan 2004 20:26:27 -0500
To: "Hector Santos" <[email-masked]>
Sender: [email-masked]

I could reply to your reply in detail but that would only distract from 
the important points in this debate.

the points are these:

if widely adopted CBV will impair the reliability of the mail system by 
causing delivery failures for legitimate messages in cases where 
delivery would otherwise have been successful (i.e. if the return-path 
field was invalid or not usable due to a configuration error or 
temporary failure)

since CBV is easily circumvented by spammers, in the long term CBV does 
no good - it only introduces overhead.

From: "Hector Santos" <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>, "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for disabled commands
Date: Thu, 15 Jan 2004 16:52:11 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

CBV - Callback Verifier.

It began in the Modem Dialup BBS days when the telephone caller id (CID) was
made possible from the phone company and compliant modems.   The CID is
provided between the first and second RING.

When the user dialed up,  CBV software performed various "call backs" to
validate the user in some form or fashion.   The simplest validation was to
ask for the phone number and compare it against the CID.  More advance
validations actually performed a call back to the CID.  For our Wildcat!
software, there are hundreds of modem based CBV 3rd party software.

When the internet came around, some of the authors continued the concept to
periodically validate email addresses usually by sending an email to the
user when they connected to the BBS.

In terms of SMTP,  when the sender issues the MAIL FROM: return path,   a
CBV will take this return path and perform a similar SMTP based callback as
if it was going to send a reply.  But it just go as far as the RCPT TO:
state to see if the return path is bad.  If bad, then we have a 100% true
negative.  If the remote says it is GOOD,  well,  you accept it and move on.

When I first started my anti-spam efforts, I was seeing that some people
(unfamiliar with older systems, including idiotic patent lawyers trying to
lay claim to a call back system) were calling this a "Challenge Response"
system.   I would not call it that.  It is not a challenge in the
traditional sense.  It is much closer to a traditional dynamic caller id CBV
concept.  For the legal eagles, this "business method" has been in place
since the day the CID was made possible with the phone companies.
Conceptually,  the SMTP protocol  client connection IP is equivalent.
However, to perform the callback it has to be based on the IP of the email
domain host.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Sent: Thursday, January 15, 2004 4:03 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for
disabled commands

>
> I am trying to follow this thread, but do not know what CBV stands
> for.  Could someone enlighten me?  Thank you.
>
> Rich Hammer
>
>

Date: Thu, 15 Jan 2004 22:48:49 +0100
From: "Rolf E. Sonneveld" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for disabled commands
To: "Richard O. Hammer" <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Reply-to: [email-masked]
Organization: Sonnection B.V.
Sender: [email-masked]

Hi, Richard,

Richard O. Hammer wrote:

>
> I am trying to follow this thread, but do not know what CBV stands 
> for.  Could someone enlighten me?  Thank you. 

Call Back Verification.

/rolf

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Thu, 15 Jan 2004 16:21:23 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

> > The bad news is that in the end, they will be legitimate (CAN-SPAM)
> > making a CBV system a redundancy issue and more of a auditing/tracking
> > system.
>
> Unless CAN-SPAM (both laws and enforcement, which is much harder)
> spreads to the whole planet, the spammers will just move offshore
> (maybe not the same folks, but there will still be spammers), and
> they'll continue to forge adddresses (legitimate ones).  So CBV will be
> worse than useless.  (which is why I didn't encourage people to use it-
> as in the long run it would only increase complexity and overhead and
> it wouldn't benefit anyone.)

That depends on the implementation.   Sender-machine validation is still
important which is what the ASRG is pretty much focused on.

I would like to STRESS I am not bent on any idea whatsoever.  Our main goal
was to provide the compatibility, be ready for new "access control"
proposals.    So we provided hooks into each state point.  The CBV turned
out to be something of a "what if" to see how we can control access based on
the current SMTP protocol.  The day there is "something" better, it is
easily switched.

Personally,  if I had my crutters,  I would prefer a strong two way
handshaking system between the server and client defining the level of
conformance,  product code concepts, caller id (ip) restrictions, time
restrictions, direct vs hub restrictions, etc.  similar to the old Fidonet
FTN (FTSC1/WAZOO) protocols.   But this is not reality.

I am not so skeptical of the laws being advocated.  I believe the US model
will be used globally.  I also personally believe  the act did its best to
basically say there is only a few things the industry ask for without
completely wiping out the "right to do business," and that is, "Please do
not lie about your return addresses and the topic."    But this is my
opinion on it.

Yes, I agree the advantage is for the spammer.  With that said, they don't
need to go off-shore.  The Act strengthens there legal position in the
states.   You are already started to see direct marketers highlight these
very points:

            "We are CAN-SPAM Compliant......
            The CAN-SPAM Act supercedes state laws....
             If you have any question, contact our legal department..."

> > Use a NULL address.
>
> That's fine if systems don't reject the address out-of-hand.  I suspect
> some still do.  My verifier tries to use a NULL address but if MAIL FROM
> is rejected, it switches to a non-NULL address.  (if the mailer rejects
> RCPT TO because FROM was NULL then the sender is out of luck) but I'm
> only verifying list traffic and I'm assuming that the other end isn't
> doing CBV at SMTP time because up to now it's been a fairly rare
> practice.

I still have to measure the NULL address rejection. See my response to Rolf
where I did some stats on this.  It was higher than I thought, but still
small.

> Of course, systems that do such checking are broken. Nothing says that a
> NULL address signals a DSN.

Yes, I agree.   But what does a NULL address signal?   Yet another
clarification?   Can a BOUNCE be sent based on the RFC822 header information
even though the ENVELOPE indicated a NULL return path?

> > Why not?
>
> Because there are *lots* of cases where you want reports to go to some
> other address than that of the person who sent the message.

I hope we are not confusing "the author" of the message.  I am certainly not
trying to imply the author is involved here.

First,  the MAIL FROM: return path must remain consistent thoughout an
entire route and only at the final destination, does the SMTP-receiver
prepend the Return-Path: header to the stored message.    Is this not
correct?

Second, regardless of what is used in the MAIL FROM:,  it still needs to be
valid.  Is this not correct?

So what if a mailing list is using some other return path for its error
reporting.  It still needs to be valid.

I've seen systems use a NULL in this case to avoid bounces from
mailer-daemons or use an alias to consolidate or redirect bounces for
further processing (i.e, auto removals, if possible).

But if its a NON-NULL return path,  the RFC presumes it must be an existing
and valid address.  That's all I am saying.

> Well, sure the address is supposed to exist (or be null), because we
> don't want to waste the resources of the mail system delivering reports
> to nonexistent addresses.  But that doesn't mean that it's the identity
> of the sender.

I believe the goal of a "CBV" is to validate the return path regardless of
who it belongs too.   In a way, the CBV is more of a SMTP compliancy tester.

> > But this does not a reason to throw out the baby with the bath water.
> > The CBV system works to eliminate  a majority of the spoofing
> > spammers.
>
> Not in the long term.  At best it's a stopgap measure.  I give it six
> months to a year.

Hehehehe.  Hard to argue.  If one is to believe there will be a positive
effect with the advent of "CAN-SPAM Ready" spammers (the marketing has
already started), then the best a CBV can offer is auditing and tracking.

> I see it as more of a configuration issue.  The protocol doesn't have
> to change to make this work.

Ok.  Be nice to see some clarification of your thought :-)

> It sounds like you're defining CBV performance in terms of CBV.  The
> relevant questions are different.  How many of those messages were
> actually valid messages that should have gone to the recipient even
> if the return-path was screwed up?

Again, the complexity of the problem was reduce to SMTP compliancy.  This is
really the only way to look at it.   Once we begin to introduce
"interpretation of the mail content", IMO, we will be beating a dead horse.
In this case, you might as well just let the data to be received and perform
delay validation.

In my view, spam is just a symptom of the real problem - anonymous access
exploitation of the SMTP functional specification which begins with the
ambiguity of various parts of the spec, hence making the technical
implementations weak in enforcing compliance.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Thu, 15 Jan 2004 16:03:24 -0500
From: "Richard O. Hammer" <[email-masked]>
To: IETF-SMTP <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for disabled commands
Sender: [email-masked]

I am trying to follow this thread, but do not know what CBV stands 
for.  Could someone enlighten me?  Thank you.

Rich Hammer

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
Date: Thu, 15 Jan 2004 15:35:23 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, January 14, 2004 10:37 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

>> Have you ruled out the possibility that some popular spamming software
simply
>> gets confused and blows chunks if it sees a 220 banner in a funny format?

Sure I considered it.    I did not rule it out.

A final test would of been to the put the words in a one line 220 response.
At the moment, the main line is not configurable, just extra lines.

But whose is at fault here?  I hate throwing the phrase around,  but their
"software is broken."  <g>

These is the type of thing I am talking about.  We need to enforce policy
(allow software to be more restrictive) if we are going even begin to see
any success in controlling the abuse of the transport system.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper reply code for disabled commands
Date: Thu, 15 Jan 2004 15:20:26 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Rolf E. Sonneveld" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, January 14, 2004 5:40 PM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

> Read Keith' answer again: he states: 'it is not reasonable to take
> Return-path as an indicator of _sender_ identity'. And you are talking
> about Return-path as the place to send notifications to. These are two
> different things:
>
> 1. a mail address to send notifications to
> 2. a sender's identity
>
> See also RFC2821:

Hi Rolf,

hmmmmm?  I hope I did not express the idea that I was referring to the
"author address?"

The author of the message plays no role in the CBV process.  The Sender or
"messenger" expressed with the Mail From: return path is the entity involved
here.

> So the Return-Path may change on the way to the recipient and is assumed
> to be set by the MTA which does the final delivery. This shows you that
> chances are high that the Return-Path header will not contain the
> original Sender address and as such cannot be used for sender
> verification purposes.

Hmmmmm, unless I am wrong,  I was under the current/modern SMTP design
assumption the initiating "Message Sender" must remain constant throughout
the entire multi-host route and that the Return-Path: header is only added
during final delivery.     This has to be true in order for the final
delivery to stamp the proper Return Path header with the proper return path.

Am I wrong?

As a side note,  am I correct in saying the mix terminology of "message
sender" vs "message originator" as used here in '2821' section 4.4 para 6,
is somewhat conflictive?

4.4 Trace Information
       ...

   It is possible for the mailbox in the return path to be different
   from the actual sender's mailbox, for example, if error responses are
   to be delivered to a special error handling mailbox rather than to
   the message sender.  When mailing lists are involved, this
   arrangement is common and useful as a means of directing errors to
   the list maintainer rather than the message originator.

I believe in technical discussions with others, "message sender" is agreed
to be the MTA (SMTP-client) transporting the message.  Although most people
clearly understand the difference and the intent of the paragraph, it might
help if the paragraph is cleaned up a bit.  Is this correct?  Am I still
confusing something?

Further, this paragraph:

   The primary purpose of the Return-path is to designate the address to
   which messages indicating non-delivery or other mail system failures
   are to be sent.  For this to be unambiguous, exactly one return path
   SHOULD be present when the message is delivered.  Systems using RFC
   822 syntax with non-SMTP transports SHOULD designate an unambiguous
   address, associated with the transport envelope, to which error
   reports (e.g., non-delivery messages) should be sent.

I believe the statement clearly presumes that an unambiguous return-path
MUST (not SHOULD) be valid in order for  the delivery of  a"system failure"
message can be attempted.  The idea an attempt will be made can only suggest
that the presumption is one of validity.   Whether it is realistically valid
or not is not the question,  the presumption is, that is it valid in order
for the attempt to take place.  Otherwise, what the primary purpose of the
return-path?  It's value is reduced.

> AFAIK MS Exchange does verify the syntax of an address during SMTP
> transmission, but _not_ the existence of the recipient address.

I believe this is a matter of implementation as it is possible to hook into
the RCPT TO: state.

> The number of Exchange servers, connected directly to the Internet is
> growing :-( As a side note: due to dictionary attacks mail system
> managers will configure their mailservers (as far as the MTA supports
> it) to always give a meaningless answer to the RCPT TO: command.

Well, IMO, I believe it is a red herring in the spammer world.  I believe
spammers are more interested in reaching the data state and sending the
message regardless of what happens from that point.  They have no control of
what happens once the data is received.  If we are to believe AOL.COM's
claim as the largest database of email users, then why is it they do dynamic
validation of the recipient address?   Why don't they delay the process? I
believe they have concluded it will make the problem worst  by lowering the
value of their end-user addresses making it possible for fraudulent usage.
In any case, IMO,  a system not validating local recipient address helps
SPAMMERS, not deter them in any way and it adds more burden and cost to the
rest of the industry by given them a database of spoofable addresses.

> I contacted them for this issue (as they provide their
> customers with an option to violate the standards) and they were not
> willing to change this configuration option. From my postmaster duties I
> know that 1-2 percent of all external mail systems run IMail. And I've
> seen other mailers rejecting NULL envelope From addresses.

Let me do a quick  stats count. This is from our own logs since
November 23, 2003 (the start of our
CBV project):

Statistics: November, 2003 day 26 to 30

wcsap calls        :    6834
skips              :    4204
-bad domain        :     323
-no mx records     :       0   (old log did not record mx total)
-bad connects      :     583  (includes errors on multiple mx)
smtp sessions      :    1981
-refused welcome   :      12
-refused noop      :       0
-refused helo      :       1
-refused mail from :     129 ( 6.51%)
-refused rcpt to   :    1060 (53.51%)

Statistics: December, 2003 day 1 to 31

wcsap calls        :   47782
skips              :   19615
-bad domain        :    2808
-no mx records     :     829
-bad connects      :    6749
smtp sessions      :   20125
-refused welcome   :     350
-refused noop      :    1012
-refused helo      :       5
-refused mail from :     885 ( 4.40%)
-refused rcpt to   :   15374 (76.39%)

Statistics: January, 2004 day 1 to 14

wcsap calls        :    4972
skips              :    3864
-bad domain        :     218
-no mx records     :     103
-bad connects      :     247
smtp sessions      :     843
-refused welcome   :      11
-refused noop      :      49
-refused helo      :       0
-refused mail from :      25 ( 2.97%)
-refused rcpt to   :     785 (93.12%)

The refused mail from  was more than I thought, but still negligible
overall. These are null addresses.  The skips are other preliminary checks
before the actual CBV is done.  But more importantly look on how high that
that RCPT TO
rejection is!    This is something that can not be ignored.    We have a
proof of concept.  :-)

> FYI: the current SPF draft claims to be a superset of RMX and DMP. See
> http://spf.pobox.com/intro.html. I agree with you that SPF has its flaws.

SPF reduces the DNS lookups to one as oppose to one or two lookups (and
possibly more with a DMP=ALTDOMAIN alternative domain response) with the DMP
proposal.     However,  they all suffer from non-compliant systems.   The
process residence time increased tremendously with high DNS lookup delays.
So what I did was provide a table of DMP domains the admin can setup for
these special lookups (local domains and known compliant systems) for now.
It helps to eliminate the local domain spoofing quite nicely.

> The main problem in my view is that everybody tries to find his/her
> solution to the SPAM problem, proposing all sorts of extensions to SMTP
> or to the way MTA's handle mail, but there is no single coordinated task
> force taking responsibility to start a _fundamental_ discussion about
> how to solve/minimize this problem. Many proposed solutions are
> solutions for an ideal world and don't take into account the real
> messaging world of today (with a myriad of possible usages of e-mail).
> E.g. the SPF folks assume that everybody will change the way they are
> doing mail forwarding.

I agree with you 100%.

In any case, although the problem is complex, I believe the scope can be
reduced to a technical SMTP level.  I truly believe that the SPAM problem is
just a symptom with the real problem that we need to deal with, anonymous
access.
Technically, I believe SMTP protocol is sound.   Developers need stronger
functional specs in order to better enforce anonymous access controls.
Yes, the forefathers will say "the internet is built on relaxed
restrictions.  you can't enforce anything."   This should not take away the
idea that we should clarify what is expected in the functional
specifications.

If the HELO can be VALIDATED, then we should say it MAY be validated and a
transaction can be stopped for non-compliance.  We shouldn't say "you must
not reject mail for failed validations."     We need to clearly define what
machine domain must be.

     HELO must a FQDN and a domain literal for dynamic IP machines.
     It MUST match the connection IP address.

Once stated, responsible developers will comply.  Commercial User agents
will change their wares to comply.

The same applies to the MAIL FROM: return path.

Either its valid or its not.  It can't be both ways.  Whether we agree with
CAN-SPAM or not, it is here. That is a reality.  Valid return paths are part
of the its mandate.  At a minimum RFC '3821' should be made consistent with
what the law now expects and enforces on spammers, whether they support it
or not.   All I am saying by this is that the RFC should make it very
possible that new future implementations CAN and MAY enforce strong
restrictions based on validating machines and the return path at the
protocol level.   As it is right now, the loop holes exist in the functional
specification.

We close the loopholes in the functional specifications and we will begin to
immediately minimize the anonymous access
problem.  A status quo position will make it fundamentally difficult for any
solution to work.   Efforts will move in different directions.

How it is done?

Well, I suggest for backward compatibility that we begin by using extended
EHLO attributes and commands to help define the level of SMTP conformity of
the sender and receiver.  i.e;

    C:  EHLO  machine-domain
    S: 250-winserver.com, Pleased to meet you.
    S: 250-RFC3821
    S: 250-SIZE 5120000
    S: 250-ETRN
    S: 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
    S: 250-AUTH=LOGIN
    S: 250 HELP

Possibly, RFC3821 can also be a tag in the greeting.

    220 xxxxxx,  ESMTP-RFC3821 [product info] server ready.

Either way, this  extended attribute begins to inform the client (regardless
if they are reading it or not) that the new RFC3821 policies are in effect
for the session.   RFC3821 offers stronger anonymous access controls a
implementation may impose consistent with the intentions of the
specifications with no ambiguity.

Of course, implementations have design it anyway they like - relaxed or
strong for non-compliant clients.  But my main point is that we give the
power back to the developer to stop the abuse.

I have an idea that is based on using the MX record to define the SMTP level
of conformity.  Why not use a subdomain naming syntax, i.e.,  (note, for
illustration purposes only)

            mx1-SMTP3.example.com

Or,  lets use the preference value + 1000 to signify a SMTP3 system.
Whatever is done,  the goal is to give current systems with little change in
their software a immediate and fast method to provide information expressing
the level of conformance the target host may have without any extra lookups.

In any case, currently I think any solution that is sought are all weaken by
the fact the RFC is fundamentally inconsistent with the efforts being made.
This thread pretty much proves my point.  We seem to be stuck on a time
warp.   Anonymous access is a problem that must be addressed sooner than
later.  :-)   As far as the CBV is concern,  the unfortunate part about it
is that it works all too well from a proof of concept point of view in
eliminating 80% or so the spoofers.  Whether CBV is the method of choose is
not the issue. If something better arrives, it is be used.  But CBV has
high-lighted that by using SMTP conformity, we can eliminate a tremendous
amount of the abuse in the mail transport system.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Thu, 15 Jan 2004 09:15:06 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
To: [email-masked]
Sender: [email-masked]

On Wed, 14 Jan 2004 [email-masked] wrote:

    > > Very little spammers use a NULL address because it
    > > reinforces further DSN checking that will fail their transaction.
    >
    > Of course, systems that do such checking are broken. Nothing says that a
    > NULL address signals a DSN.

    And in fact, LSoft's Listserv product does this.  It will send
    things like cookie confirmations for subscription requests with a
    MAIL FROM:<>...

And I'll go a step further.  I've been doing this for years (way before
spam was the problem it is today) not only with confirmation requests
but almost all auto-responder messages.

You commented about the failed message being useless but more to the
point, just what action would you take if you got one?  I suppose you
could log it but to what end?

I've always used the NULL MAIL FROM: because to do otherwise
unnecessarily uses resources.

Jim

To: Keith Moore <[email-masked]>
Cc: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
From: [email-masked]
Date: Wed, 14 Jan 2004 22:41:46 -0500
Sender: [email-masked]

--==_Exmh_-1576434044P
Content-Type: text/plain; charset=us-ascii

On Wed, 14 Jan 2004 17:40:15 EST, Keith Moore said:

> > Very little spammers use a NULL address because it
> > reinforces further DSN checking that will fail their transaction. 
> 
> Of course, systems that do such checking are broken. Nothing says that a
> NULL address signals a DSN.

And in fact, LSoft's Listserv product does this.  It will send things like cookie
confirmations for subscription requests with a MAIL FROM:<> specifically
so that if the request came from a duff address, the Listserv reply goes out
and bounces, it won't get a totally useless bounce message coming back at it.

--==_Exmh_-1576434044P
Content-Type: application/pgp-signature

--==_Exmh_-1576434044P--

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
From: [email-masked]
Date: Wed, 14 Jan 2004 22:37:10 -0500
Sender: [email-masked]

--==_Exmh_-1579413668P
Content-Type: text/plain; charset=us-ascii

On Wed, 14 Jan 2004 16:30:02 EST, Hector Santos <[email-masked]>  said:

> More ironically, what I have seen is a 60% increase of HELO drops once we
> place a POLICY STATEMENT in the greeting with the specific words:
> 
>         WARNING:  FOR AUTHORIZED USE ONLY!
> 
> I believe this is result of the AOL lawsuit.  So the BIG spammers do look
> for these SPECIFIC words and will drop their connections once seen.

Have you ruled out the possibility that some popular spamming software simply
gets confused and blows chunks if it sees a 220 banner in a funny format?

--==_Exmh_-1579413668P
Content-Type: application/pgp-signature

--==_Exmh_-1579413668P--

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands 
From: [email-masked]
Date: Wed, 14 Jan 2004 22:33:55 -0500
Sender: [email-masked]

--==_Exmh_-1581846572P
Content-Type: text/plain; charset=us-ascii

On Wed, 14 Jan 2004 16:30:02 EST, Hector Santos <[email-masked]>  said:

> > Return-path is intended for (non)delivery notification, not for sender
> > verification.  It is not reasonable to take return-path as an indicator
> > of sender identity.
> 
> Why not?  I believe the presumption is made in RFC 1123 and in RFC 2821 that
> DNS and error reporting is to be returned to a "existing" return path.
> Otherwise what is the point of using a Return-Path: header?

*exactly*.  Return-Path (which is derived from the 2821 MAIL FROM) says
*where to send the bounces*.  As such, it tells you almost nothing about who
actually *sent* it - consider that the mail I'm replying to would be considered
as being "sent" by either Hector Santos, or the ietf-smtp mailing list, depending
on your religion.  owner-ietf-smtp is where bounces go, not who the sender is.

I have a home-grown "probe for dead list owners" script for our Listserv system.
It sends mail with an RFC2822 From: of [email-masked]', but the
2821 MAIL FROM: has [email-masked]', which, not surprisingly
feeds into a Perl program that does the first-pass parsing of bounces for me.

Did dead-owners send the mail? No, it's just catching the bounces.

--==_Exmh_-1581846572P
Content-Type: application/pgp-signature

--==_Exmh_-1581846572P--

Date: Wed, 14 Jan 2004 17:40:15 -0500
From: Keith Moore <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Sender: [email-masked]

> > The reason I asked is that I used CBV for many years to filter spam
> > from my mailing lists, and I often found that (otherwise) legitimate
> > addresses failed to verify if I used MAIL FROM:<>.  (I was using CBV
> > on the header From address - on the assumption that if someone on
> > the list couldn't reply to the message by normal means, then it
> > probably wasn't useful to send that non-subscriber's comments to the
> > list anyway).
> >
> > Most of the time the verification would fail in the MAIL FROM
> > response, but occasionally it would fail in RCPT TO response.  So I
> > ended up having to send MAIL FROM:<[email-masked]> or some such
> > instead in order to get an accurate verification.
> >
> > (one could argue that you shoudn't accept mail from misconfigured
> > systems anyway, but bouncing such messages doesn't do anything
> > useful IMHO because it doesn't report an error that is likely to be
> > understood by the person receiving the bounced mail.  That and you
> > have to arrange for the bounces to have a return-path other than <>
> > in order for them to even get back to the sender.)
> 
> I'm not seeng the same level of result you seem to suggest occurs in
> mass. Yes, some systems will behave differently when provided a NULL
> vs non-NULL Mail From:   but it is a very relative few from what I
> see.

The situation may have changed in the last year (it would certainly be
good if MTA admins were getting smarter).  Or it might be that we're
testing different user communities.  (one of the odd things about spam
detection is that it's very hard to get a representative sample - just a
method works well with one sample doesn't mean it will work well with
another.  And a large sample is not necessarily a good indicator of how
well a small sample will behave.)

> But the goal of the CBV, at least ours, is to check for a BAD address.
> 

same for mine.
 
> The good news is that the BAD systems are the ones mostly likely
> providing the results we are looking for.   By far, they are spoofing
> fake addresses and this is what we are helping to eliminate in this
> process.

not in my experience.

> In other words, a CBV is catching the majority of the spoofing
> spammers.

not in my experience.
 
> The bad news is that in the end, they will be legitimate (CAN-SPAM)
> making a CBV system a redundancy issue and more of a auditing/tracking
> system.

Unless CAN-SPAM (both laws and enforcement, which is much harder)
spreads to the whole planet, the spammers will just move offshore
(maybe not the same folks, but there will still be spammers), and
they'll continue to forge adddresses (legitimate ones).  So CBV will be
worse than useless.  (which is why I didn't encourage people to use it-
as in the long run it would only increase complexity and overhead and
it wouldn't benefit anyone.)

> 
> > Here's what I see happening if this is widely adopted:
> >
> > A0->B0: MAIL FROM:<my.address>
> > -B0->A1: HELO
> > -B0->A1: MAIL FROM:<dummy.address@B>
> > --A1->B1: HELO
> > --A1->B1: MAIL FROM:<dummy.address@A>
> > ---B1->A2: HELO
> > ---B1->A2:MAIL FROM:<dummy.address@B>
> > ----A2->B2: HELO
> > ----A2->B2: MAIL FROM:<dummy.address@A>
> > ...
> 
> Use a NULL address. 

That's fine if systems don't reject the address out-of-hand.  I suspect
some still do.  My verifier tries to use a NULL address but if MAIL FROM
is rejected, it switches to a non-NULL address.  (if the mailer rejects
RCPT TO because FROM was NULL then the sender is out of luck) but I'm
only verifying list traffic and I'm assuming that the other end isn't
doing CBV at SMTP time because up to now it's been a fairly rare
practice.

> Very little spammers use a NULL address because it
> reinforces further DSN checking that will fail their transaction. 

Of course, systems that do such checking are broken. Nothing says that a
NULL address signals a DSN.

> > > What is the RFC 2821 presumption for the validity of the return
> > > path?
> > >
> > > Or at what point should RFC 2821 presume the return path to be
> > > valid? When
> > > it is provided or when its too late (after the mail is received
> > > and rejected)?
> >
> > Return-path is intended for (non)delivery notification, not for
> > sender verification.  It is not reasonable to take return-path as an
> > indicator of sender identity.
> 
> Why not?  

Because there are *lots* of cases where you want reports to go to some 
other address than that of the person who sent the message.

> I believe the presumption is made in RFC 1123 and in RFC
> 2821 that DNS and error reporting is to be returned to a "existing"
> return path. 

Well, sure the address is supposed to exist (or be null), because we
don't want to waste the resources of the mail system delivering reports
to nonexistent addresses.  But that doesn't mean that it's the identity
of the sender.

> > SMTP servers _should_ return immediate error conditions (in response
> > to MAIL FROM, RCPT TO, DATA) whenever possible because, for a wide
> > variety of reasons, immediate reporting is much more reliable and
> > consumes less resources than bouncing the message.
> 
> I agree.   But there are still legacy and legimiate reasons for
> delaying the verificaton/validation/checking at RCPT TO:

Yes, though we'd do well to discourage it.  Some of the reasons are
good, others are bogus.  IMHO.

> In our case, the CBV does not work well for a YAHOO.COM type system
> because it is not reaching the DATA state.  I personally believe YAHOO
> acceptance of all recipients contributes the the spamming problem and
> distribute the cost to other systems because spoofers will use
> YAHOO.COM as the return domain knowing YAHOO.COM will accept it.

could be.

> But this does not a reason to throw out the baby with the bath water. 
> The CBV system works to eliminate  a majority of the spoofing
> spammers.

Not in the long term.  At best it's a stopgap measure.  I give it six
months to a year.  

> > For similar reasons, SMTP senders
> > _should_ minimize the relay path length to the receiver's MX rather
> > than aggregating traffic through upstream relays.  Of course there
> > are reasons to have both outbound and inbound relays, but these do
> > come at a cost.
> 
> Again I agree.   Keep in mind that this is will alter the
> specifications, hencing making a deployment issue.

I see it as more of a configuration issue.  The protocol doesn't have
to change to make this work.
 
> > Despite my use of CBV on mailing list traffic, I don't think they
> > are nearly as good an indicator for traffic in general, for lots of
> > reasons:
> >
> > - The presumption that traffic with an invalid return-path is
> > invalid, while perhaps reasonable for some mailing lists, is less
> > reasonable as a general rule.
> 
> Emperically,  I am not finding this to be true.  With over 25 testing
> sites processing millions of messages (for our systems an average of
> 2500+ or so per day),   +80% of the transactions are rejected due to
> CBV checking with 100% TRUE negatives (remotes will refuse the return
> address when the RCPT TO is issued at their site).

It sounds like you're defining CBV performance in terms of CBV.  The
relevant questions are different.  How many of those messages were
actually valid messages that should have gone to the recipient even
if the return-path was screwed up?  Perhaps the sender mistyped his
domain name into his user agent preferences and was trying to send
mail to his tech support staff.  Perhaps the DNS administrator put the
wrong MX into the DNS zone, or forgot to update the serial number
when editing the zone file, so the mail is still going to the old MX
or A record instead of the new one.  Perhaps the SMTP server was
misconfigured, or perhaps it uses NIS to verify addresses and the NIS
server is down and the MTA isn't able to distinuish temporary NIS errors
from "user does not exist".  I've seen all of these happen many times.

> > - The verification has lots of overhead, and some potential for
> > looping if widely adopted.
> 
> A technical problem that can be addressed and has been addressed.  A
> NULL address is all you need to do. 

My experience is different, though it is somewhat dated.
 
> > - The effectiveness of CBV has decreased considerably over time. 
> > When I first started using it (not sure when, maybe 7 years ago) it
> > identified nearly 100% of spam sent to my lists.  When I stopped
> > using it on a large scale (just over a year ago) it only identified
> > about 25% of the spam sent to my lists.  It appeared that spammers
> > had realized that mail from invalid addresses was less successful,
> > so they simply started sending mail from valid addresses.
> 
> Again, today, we are seeing atleast 80% of spammers are spoofers.  I
> believe industry research also agrees with these level of numbers and
> I believe the main reasin why it is one of the mandates of CAN-SPAM
> Act; validate return addresses.

Wasted effort.  Spammers will circumvent it.

> Now, I will say that I expected the same results; with Spammers
> learning. Emperically, I have not seen this. 

I can't say for sure why the effectivenss of CBV dropped.  I can only
say that it did drop.

> I believe this is
> because it is MORE in their interest to remain anonymous than to try
> issue valid but fraudulent use of addresses which is a ECPA federal
> crime.

I do think that spammers have learned to avoid using addresses that
other people are using.  I'm not getting many complaints that I'm
spamming  people anymore (resulting from spammers forging my address). 
But the return addresses on most of the spam I see are verifiable using
SMTP. Whether they can be reliably associated with the spammer, I cannot
say.

> More ironically, what I have seen is a 60% increase of HELO drops once
> we place a POLICY STATEMENT in the greeting with the specific words:
> 
>         WARNING:  FOR AUTHORIZED USE ONLY!
> 
> I believe this is result of the AOL lawsuit.  So the BIG spammers do
> look for these SPECIFIC words and will drop their connections once
> seen.
> 
> I personally couldn't believe my eyes.  I remove the policy statement
> and behold, they didn't drop.  Put back, they began to drop again.

At least that's a low-overhead solution :)

> But I have not seen them learn the MAIL FROM.
> 
> What I have seen is some of them will issue MULTIPLE MAIL FROMS: if
> dynamically rejected.   I seen atleast 2-3 attempts here before
> dropping.

Maybe you're seeing my verifier :)
 
> > - In my experience, CBV does not provide 100% true negatives -
> > because of misconfigured systems that reject MAIL FROM:<> and other
> > valid mail.
> 
> When I say 100% true negatives, I am talking about 100% rejections at
> the remote RCPT TO state.  That is, IMO, a 100% trusted value.

My experience is different than yours.  
 
> I can give you 6 months worth of millions of trace logs. I will be
> surprise to see more than 0.01% of MAIL FROM: <> breaks.  I will try
> to find this count for you today.
> 
> >   (if you have your SMTP server report 4xx whenever the CBV
> >   tempfails,
> > there are other problems - it is not unusual for a CBV to fail
> > because either the sender's DNS servers or the sender's SMTP server
> > are too slow or inaccessible, even though the sender's system is
> > sending outgoing mail.)
> 
> From a SMTP compatibility standpoint, this is the one of the main
> issues. Fortunately, emperically, what we are seeing is that by far
> MOST systems are SMTP compliant, run good systems and the onces the
> CBV will catch are bad systems and for the false negatives, the 45x
> does tell the remote "Hey, try again"   Fortunately, the bad systems
> WILL not.  The good systems will. Thats what I am seeing.
> 
> But again, we should not throw out the baby with the bath water.

This isn't a baby.  A baby is valuable because of its future potential,
and CBV has a very limited future potential.

> > In short - CBV is expensive for the receiver and easily defeated by
> > spammers.
> 
> By becoming legit and trackable?  I see that as a good thing.

Just because you can check a return-path with RCPT doesn't mean this
helps you trace the sender.  RCPT tells you nothing about whether
the sender actually owns that address.  We can't assume that spammers
are law-abiding.

> >   widely adopted it very quickly becomes useless overhead.
> 
> Rather useless redundancy.  Again, a good thing.  Not a bad thing.

Incurring overhead for no useful purpose is at best wasteful and 
expensive, and at worst an invitation to denial-of-service attacks.

> > I do think we need reliable traceability to the sender, but doing
> > CBV on return-path is not the way to get it.
> 
> Then you have a Deployment problem and you will not be able to do it
> without CHANGE.

Indeed, this is true.
 
> Something has to give.  Either you work with the given specs or you
> change the specs, in which case, you will need a central authority
> system because SPAMMERS are not going to change othewise.

It doesn't have to be central.
 
> > CAN-SPAM is a crock.  We should not presume that the US Congress is
> > qualified to dictate technical standards - heck, they can't even
> > write nontechnical laws that are in the public interest.
> 
> In any case,  the mandate is reality:
> 
> - Valid Return Addresses
> - Topic Indentication.

When people use the word "reality" what they generally tend to mean is
"according to my prejudice".  CAN-SPAM may influence US-based spammers
but that doesn't mean it will get rid of spam.

> 
> > > 2) Mr. Crocker's ASRG Proposal Guideline Document,
> > > draft-crocker-spam-techconsider-02.txt, emphasizes incremental and
> > > backward compatibility minimizes or quelling any desire to
> > > fundamentally
> alter
> > > the SMTP protocol.
> >
> > I haven't read this document yet, so I can't comment on the extent
> > to which I think the argument in the document is valid.  But using
> > return-path as a sender identity conflicts with its purpose in SMTP
> > as the destination of (non)delivery reports.  The Sender field from
> > RFC 822 is the closest thing to a verifiable sender identity from
> > the original design, but it is so widely misused today that it is
> > not salvageable. Basically the requirements for verification are
> > such that we're going to need a new field.
> 
> Whatever is done, we need to address anonymous access.

I agree - we need to make sure that anonymous access is provided for.
Of course that doesn't mean people have to accept mail from anonymous
senders.

> 3 parameters are currently available:
> 
> CIP - Connection Address
> CDN - Client domain name
> RP - Return Path

All of these are insufficient, for various reasons.  We're not going
to solve the spam problem without protocol extensions.

> The point is,  either the bounce address is require or not.  If so,
> then we need to make it a stronger part of the specification and then
> FROM there we can design a better, optimized system.  

What you are proposing to do is to change what the bounce address is 
used for, which isn't the same thing as strengthening it.

> All am saying less finally do something about it. 

...whether it makes sense or not?

> Lets not say its impossible to solve. 

It's not impossible to solve.  However, that doesn't mean that the
solution is attractive.  For instance, do we really want to make it
easy for governments to trace every single email message to the sender?
Because it's very hard to build a system that lets spammers be 
identified reliably (and held accountable) that doesn't have the side
effect of enabling mass surveillance by governments.  Or do we really
want to have a single root CA for all email?  That has its own problems.

At the same time, pretending that we can solve the spam problem without
significant changes is just wishful thinking - there's no basis for
believing that it will work.

Keith

Date: Wed, 14 Jan 2004 23:40:20 +0100
From: "Rolf E. Sonneveld" <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
To: Hector Santos <[email-masked]>
Cc: [email-masked]
Reply-to: [email-masked]
Organization: Sonnection B.V.
Sender: [email-masked]

Hector Santos wrote:

[...]

>>Return-path is intended for (non)delivery notification, not for sender
>>verification.  It is not reasonable to take return-path as an indicator
>>of sender identity.
>>    
>>
>
>Why not?  I believe the presumption is made in RFC 1123 and in RFC 2821 that
>DNS and error reporting is to be returned to a "existing" return path.
>Otherwise what is the point of using a Return-Path: header?
>
>I believe this needs to be made very clear.
>

Read Keith' answer again: he states: 'it is not reasonable to take 
Return-path as an indicator of _sender_ identity'. And you are talking 
about Return-path as the place to send notifications to. These are two 
different things:

1. a mail address to send notifications to
2. a sender's identity

See also RFC2821:

>    When the delivery SMTP server makes the "final delivery" of a
>    message, it inserts a return-path line at the beginning of the mail
>    data.  This use of return-path is required; mail systems MUST support
>    it.  The return-path line preserves the information in the <reverse-
>    path> from the MAIL command.  Here, final delivery means the message
>    has left the SMTP environment.  Normally, this would mean it had been
>    delivered to the destination user or an associated mail drop, but in
>    some cases it may be further processed and transmitted by another
>    mail system.
>
>    It is possible for the mailbox in the return path to be different
>    from the actual sender's mailbox, for example, if error responses are
>    to be delivered to a special error handling mailbox rather than to
>    the message sender.  When mailing lists are involved, this
>    arrangement is common and useful as a means of directing errors to
>    the list maintainer rather than the message originator.
>
>    The text above implies that the final mail data will begin with a
>    return path line, followed by one or more time stamp lines.  These
>    lines will be followed by the mail data headers and body [32].
>
>    It is sometimes difficult for an SMTP server to determine whether or
>    not it is making final delivery since forwarding or other operations
>    may occur after the message is accepted for delivery.  Consequently,
>
>
>
>
> Klensin                     Standards Track                    [Page 50]
> 
> RFC 2821             Simple Mail Transfer Protocol            April 2001
>
>
>    any further (forwarding, gateway, or relay) systems MAY remove the
>    return path and rebuild the MAIL command as needed to ensure that
>    exactly one such line appears in a delivered message.
>
>    A message-originating SMTP system SHOULD NOT send a message that
>    already contains a Return-path header.  SMTP servers performing a
>    relay function MUST NOT inspect the message data, and especially not
>    to the extent needed to determine if Return-path headers are present.
>    SMTP servers making final delivery MAY remove Return-path headers
>    before adding their own.
>
>    The primary purpose of the Return-path is to designate the address to
>    which messages indicating non-delivery or other mail system failures
>    are to be sent.  For this to be unambiguous, exactly one return path
>    SHOULD be present when the message is delivered.  Systems using RFC
>    822 syntax with non-SMTP transports SHOULD designate an unambiguous
>    address, associated with the transport envelope, to which error
>    reports (e.g., non-delivery messages) should be sent.

So the Return-Path may change on the way to the recipient and is assumed 
to be set by the MTA which does the final delivery. This shows you that 
chances are high that the Return-Path header will not contain the 
original Sender address and as such cannot be used for sender 
verification purposes.

[...]

>The problem I see here with some domain (specifically YAHOO.COM) is that
>they delay the validating of the recipient until the DATA stage.   So the
>harvesting reason may people cite for not validating at the RCPT TO: is a
>red herring IMO in the case of YAHOO.COM.
>

AFAIK MS Exchange does verify the syntax of an address during SMTP 
transmission, but _not_ the existence of the recipient address. The 
number of Exchange servers, connected directly to the Internet is 
growing :-( As a side note: due to dictionary attacks mail system 
managers will configure their mailservers (as far as the MTA supports 
it) to always give a meaningless answer to the RCPT TO: command.

>[...]
>
>I can give you 6 months worth of millions of trace logs. I will be surprise
>to see more than 0.01% of MAIL FROM: <> breaks.  I will try to find this
>count for you today.
>

I know that IMail (from Ipswitch, Inc., see http://www.ipswitch.com) 
does provide a configuration option to block the NULL envelope From 
address. I contacted them for this issue (as they provide their 
customers with an option to violate the standards) and they were not 
willing to change this configuration option. From my postmaster duties I 
know that 1-2 percent of all external mail systems run IMail. And I've 
seen other mailers rejecting NULL envelope From addresses.

[...]

>True, if EVERYONE was using it.   There will be a major DNS lookup overhead
>when he CDN and RPD does not exist and this was empirically proven in our
>testing as we tried to look for all possible ideas that can reduce the CBV
>necessity.  DMP and others, like RMX and SPF all suffer from the same
>fundamental deployment flaw.
>

FYI: the current SPF draft claims to be a superset of RMX and DMP. See 
http://spf.pobox.com/intro.html. I agree with you that SPF has its flaws.

The main problem in my view is that everybody tries to find his/her 
solution to the SPAM problem, proposing all sorts of extensions to SMTP 
or to the way MTA's handle mail, but there is no single coordinated task 
force taking responsibility to start a _fundamental_ discussion about 
how to solve/minimize this problem. Many proposed solutions are 
solutions for an ideal world and don't take into account the real 
messaging world of today (with a myriad of possible usages of e-mail). 
E.g. the SPF folks assume that everybody will change the way they are 
doing mail forwarding.

[...]

/rolf

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Wed, 14 Jan 2004 16:30:02 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Wednesday, January 14, 2004 9:47 AM
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for
disabled commands

>
> The reason I asked is that I used CBV for many years to filter spam
> from my mailing lists, and I often found that (otherwise) legitimate
> addresses failed to verify if I used MAIL FROM:<>.  (I was using CBV on
> the header From address - on the assumption that if someone on the list
> couldn't reply to the message by normal means, then it probably wasn't
> useful to send that non-subscriber's comments to the list anyway).
>
> Most of the time the verification would fail in the MAIL FROM response,
> but occasionally it would fail in RCPT TO response.  So I ended up
> having to send MAIL FROM:<[email-masked]> or some such instead in
> order to get an accurate verification.
>
> (one could argue that you shoudn't accept mail from misconfigured
> systems anyway, but bouncing such messages doesn't do anything useful
> IMHO because it doesn't report an error that is likely to be understood
> by the person receiving the bounced mail.  That and you have to arrange
> for the bounces to have a return-path other than <> in order for them
> to even get back to the sender.)

I'm not seeng the same level of result you seem to suggest occurs in mass.
Yes, some systems will behave differently when provided a NULL vs non-NULL
Mail From:   but it is a very relative few from what I see.

But the goal of the CBV, at least ours, is to check for a BAD address.  Not
Good Address and by far, most systems are compliant and most systems will
validate a RCPT TO: request.

The good news is that the BAD systems are the ones mostly likely providing
the results we are looking for.   By far, they are spoofing fake addresses
and this is what we are helping to eliminate in this process.

In other words, a CBV is catching the majority of the spoofing spammers.

The bad news is that in the end, they will be legitimate (CAN-SPAM) making a
CBV system a redundancy issue and more of a auditing/tracking system.

> Here's what I see happening if this is widely adopted:
>
> A0->B0: MAIL FROM:<my.address>
> -B0->A1: HELO
> -B0->A1: MAIL FROM:<dummy.address@B>
> --A1->B1: HELO
> --A1->B1: MAIL FROM:<dummy.address@A>
> ---B1->A2: HELO
> ---B1->A2:MAIL FROM:<dummy.address@B>
> ----A2->B2: HELO
> ----A2->B2: MAIL FROM:<dummy.address@A>
> ...

Use a NULL address. That will solve the looping problem with NO change to
the SMTP specification.  Very little spammers use a NULL address because it
reinforces further DSN checking that will fail their transaction.    In our
case,  a NULL will bypass the CBV (but not other checks).

If CBV is widely adopted then we can use some new extensions to help the
process.   But today, you have no choice, but to use a NULL address.
However in our system, it offers session parameter configuration on an
optional per-domain basis.  So if the sysadmin needs to verify a specific
domain (HOTMAIL.COM), then I believe, off hand, this domain requires a
non-NULL address.  I don't remember off-hand if hotmail.com was one of them.

Thus far, there are 3 systems that I know support CBV systems:

       - Exim  (Spelling) ???
       - Verisign.net (or verisign.com??)
       - Wildcat! WCSAP (ours)

> > I think the more appropriate question is:
> >
> > What is the RFC 2821 presumption for the validity of the return path?
> >
> > Or at what point should RFC 2821 presume the return path to be valid?
> > When
> > it is provided or when its too late (after the mail is received and
> > rejected)?
>
> Return-path is intended for (non)delivery notification, not for sender
> verification.  It is not reasonable to take return-path as an indicator
> of sender identity.

Why not?  I believe the presumption is made in RFC 1123 and in RFC 2821 that
DNS and error reporting is to be returned to a "existing" return path.
Otherwise what is the point of using a Return-Path: header?

I believe this needs to be made very clear.

> SMTP servers _should_ return immediate error conditions (in response to
> MAIL FROM, RCPT TO, DATA) whenever possible because, for a wide variety
> of reasons, immediate reporting is much more reliable and consumes less
> resources than bouncing the message.

I agree.   But there are still legacy and legimiate reasons for delaying the
verificaton/validation/checking at RCPT TO:

The problem I see here with some domain (specifically YAHOO.COM) is that
they delay the validating of the recipient until the DATA stage.   So the
harvesting reason may people cite for not validating at the RCPT TO: is a
red herring IMO in the case of YAHOO.COM.

In our case, the CBV does not work well for a YAHOO.COM type system because
it is not reaching the DATA state.  I personally believe YAHOO acceptance of
all recipients contributes the the spamming problem and distribute the cost
to other systems because spoofers will use YAHOO.COM as the return domain
knowing YAHOO.COM will accept it.

But this does not a reason to throw out the baby with the bath water.  The
CBV system works to eliminate  a majority of the spoofing spammers.

> For similar reasons, SMTP senders
> _should_ minimize the relay path length to the receiver's MX rather
> than aggregating traffic through upstream relays.  Of course there are
> reasons to have both outbound and inbound relays, but these do come at
> a cost.

Again I agree.   Keep in mind that this is will alter the specifications,
hencing making a deployment issue.

> Despite my use of CBV on mailing list traffic, I don't think they are
> nearly as good an indicator for traffic in general, for lots of
> reasons:
>
> - The presumption that traffic with an invalid return-path is invalid,
> while perhaps reasonable for some mailing lists, is less reasonable as
> a general rule.

Emperically,  I am not finding this to be true.  With over 25 testing sites
processing millions of messages (for our systems an average of 2500+ or so
per day),   +80% of the transactions are rejected due to CBV checking with
100% TRUE negatives (remotes will refuse the return address when the RCPT TO
is issued at their site).

> - The verification has lots of overhead, and some potential for looping
> if widely adopted.

A technical problem that can be addressed and has been addressed.  A NULL
address is all you need to do.  Fortunatelly most spammers do not use a NULL
address.

> - The effectiveness of CBV has decreased considerably over time.  When
> I first started using it (not sure when, maybe 7 years ago) it
> identified nearly 100% of spam sent to my lists.  When I stopped using
> it on a large scale (just over a year ago) it only identified about 25%
> of the spam sent to my lists.  It appeared that spammers had realized
> that mail from invalid addresses was less successful, so they simply
> started sending mail from valid addresses.

Again, today, we are seeing atleast 80% of spammers are spoofers.  I believe
industry research also agrees with these level of numbers and I believe the
main reasin why it is one of the mandates of CAN-SPAM Act; validate return
addresses.

Now, I will say that I expected the same results; with Spammers learning.
Emperically, I have not seen this.  I believe this is because it is MORE in
their interest to remain anonymous than to try issue valid but fraudulent
use of addresses which is a ECPA federal crime.

More ironically, what I have seen is a 60% increase of HELO drops once we
place a POLICY STATEMENT in the greeting with the specific words:

        WARNING:  FOR AUTHORIZED USE ONLY!

I believe this is result of the AOL lawsuit.  So the BIG spammers do look
for these SPECIFIC words and will drop their connections once seen.

I personally couldn't believe my eyes.  I remove the policy statement and
behold, they didn't drop.  Put back, they began to drop again.

But I have not seen them learn the MAIL FROM.

What I have seen is some of them will issue MULTIPLE MAIL FROMS: if
dynamically rejected.   I seen atleast 2-3 attempts here before dropping.

> - In my experience, CBV does not provide 100% true negatives - because
> of misconfigured systems that reject MAIL FROM:<> and other valid mail.

When I say 100% true negatives, I am talking about 100% rejections at the
remote RCPT TO state.  That is, IMO, a 100% trusted value.

I can give you 6 months worth of millions of trace logs. I will be surprise
to see more than 0.01% of MAIL FROM: <> breaks.  I will try to find this
count for you today.

>   (if you have your SMTP server report 4xx whenever the CBV tempfails,
> there are other problems - it is not unusual for a CBV to fail because
> either the sender's DNS servers or the sender's SMTP server are too
> slow or inaccessible, even though the sender's system is sending
> outgoing mail.)

>From a SMTP compatibility standpoint, this is the one of the main issues.
Fortunately, emperically, what we are seeing is that by far MOST systems are
SMTP compliant, run good systems and the onces the CBV will catch are bad
systems and for the false negatives, the 45x does tell the remote "Hey, try
again"   Fortunately, the bad systems WILL not.  The good systems will.
Thats what I am seeing.

But again, we should not throw out the baby with the bath water.

> In short - CBV is expensive for the receiver and easily defeated by
> spammers.

By becoming legit and trackable?  I see that as a good thing.

>   widely adopted it very quickly becomes useless overhead.

Rather useless redundancy.  Again, a good thing.  Not a bad thing.

> I do think we need reliable traceability to the sender, but doing CBV
> on return-path is not the way to get it.

Then you have a Deployment problem and you will not be able to do it without
CHANGE.

Something has to give.  Either you work with the given specs or you change
the specs, in which case, you will need a central authority system because
SPAMMERS are not going to change othewise.

> CAN-SPAM is a crock.  We should not presume that the US Congress is
> qualified to dictate technical standards - heck, they can't even write
> nontechnical laws that are in the public interest.

In any case,  the mandate is reality:

- Valid Return Addresses
- Topic Indentication.

> > 2) Mr. Crocker's ASRG Proposal Guideline Document,
> > draft-crocker-spam-techconsider-02.txt, emphasizes incremental and
> > backward compatibility minimizes or quelling any desire to fundamentally
alter
> > the SMTP protocol.
>
> I haven't read this document yet, so I can't comment on the extent to
> which I think the argument in the document is valid.  But using
> return-path as a sender identity conflicts with its purpose in SMTP as
> the destination of (non)delivery reports.  The Sender field from RFC
> 822 is the closest thing to a verifiable sender identity from the
> original design, but it is so widely misused today that it is not
> salvageable. Basically the requirements for verification are such that
> we're going to need a new field.

Whatever is done, we need to address anonymous access.

3 parameters are currently available:

CIP - Connection Address
CDN - Client domain name
RP - Return Path

The ASRG is focusing on validating the sender machine using a combination of
the above.

In the DMP DNS lookup proposal for example, the  format/rule is:

TXT LOOKUP for:

            RIP.in-addr._smtp-client.CDN
            RIP.in-addr._smtp-client.RPD

where RIP is the reverse address of CIP and RPD is the return path domain.
If either return
DMP=ALLOW or DMP=DENY, then you have logic to work with.

It looks to me that ASRG is focusing on DNS lookup solutions for the
red-herring reason it offers the fastest and lowest overhead "lookup"
solution.

True, if EVERYONE was using it.   There will be a major DNS lookup overhead
when he CDN and RPD does not exist and this was empirically proven in our
testing as we tried to look for all possible ideas that can reduce the CBV
necessity.  DMP and others, like RMX and SPF all suffer from the same
fundamental deployment flaw.

DMP works nicely to validate your own local domain, so if you do a TXT
lookup for:

        8.131.247.208.in-addr._smtp-client.mail.winserver.com

you will see a DMP=ALLOW

        *.in-addr._smtp-client.mail.winserver.com

returns a DMP=DENY.

So this alone has helped stop spoofing spammers from using our own domain or
machines to gain entry.

All in all,  in short,  I am the last person who wants to add anything that
isn't necessary to the product design and backward compatibility is
important.

However, reality is reality.  Anonymous access is a MAJOR industry problem
and the exploitations of the SMTP protocol is right there in our face.  We
can sit around and do nothing about it and  allow the Mickysofts, AOL and
other dictate what the future will be.  Some like AOL, Earthlink and
RoadRunner have already begun to violate the RFC by automatically rejecting
dynamic IP machines at the connecting level.  Some people have suggested
this is not a violation because it is done at the connection level.  True,
but the fact the notification is provided at the SMTP level (55x greeting),
this makes it a RFC violation.

The point is,  either the bounce address is require or not.  If so, then we
need to make it a stronger part of the specification and then FROM there we
can design a better, optimized system.   But as it stands now, we only have
a CBV as the only possible way to get a MAJORITY of the spammers out of the
picture and in my opinion, if a CBV and CAN-SPAM makes them legitimate, then
that's all the better,  there is less reason to turn on your CBV on these
domains.  Less overhead, less redundancy.

That's my view.

All am saying less finally do something about it.  Lets not say its
impossible to solve.  In my view, it can be solved at a technical level and
by solve, I mean minimize the spamming issue to one that is 100% mail
content related and out of the SMTP decision making process.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: [email-masked]
To: [email-masked], [email-masked]
Cc: [email-masked]
Subject: RE: CAN-SPAM was RE: CBV systems - was Re: SMTP Extensions - prop er peply code for disabled commands
Date: Wed, 14 Jan 2004 09:11:21 -0600
Sender: [email-masked]

Keith Moore wrote:

CAN-SPAM is a crock.  We should not presume that the US Congress is 
qualified to dictate technical standards - heck, they can't even write 
nontechnical laws that are in the public interest.

--

I agree about CAN-SPAM.  I have seen a slight decrease during the first 13
days of this month over December in terms of the amount of SPAM we have
received.  In the last few days I am seeing the numbers increase so I'm left
to assume that people are just moving off shore.

Frankly, the US Congress isn't going to help by legislating a global system.
We have to do it ourselves either through technology or self policing.  I
would vastly prefer an extension to SMTP that allowed for the passing of an
authentication code for each e-mail sent that would have to be verified by
the receiving system.

Ed 

Cc: Keith Moore <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Wed, 14 Jan 2004 09:47:46 -0500
To: "Hector Santos" <[email-masked]>
Sender: [email-masked]

>> I assume that this CBV service uses SMTP to determine whether the
>> return-path is valid.
>
> Your presumption is correct.
>
> hmmmmmm,  Maybe I should plead the 5th?  :-)
>
>> If so, does that CBV send MAIL FROM:<>?
>
> Configurable.  However, a NULL address is highly suggested for the 
> reason
> you suspect.
>
>> If so, what happens when the SMTP server associated with the 
>> return-path
> (incorrectly) returns an
>> error to that? (in my experience this is a fairly common bug, or
>> misfeature, or misconfiguration).
>
> In our testing with over 25 commerical systems, millions of 
> transactions.  I
> believe I have personally seen logs with maybe a few of these type of
> sessions. In all cases, it was confirmed to be "zombie-like" sites that
> intentionally fail at the protocol level.

The reason I asked is that I used CBV for many years to filter spam 
from my mailing lists, and I often found that (otherwise) legitimate 
addresses failed to verify if I used MAIL FROM:<>.  (I was using CBV on 
the header From address - on the assumption that if someone on the list 
couldn't reply to the message by normal means, then it probably wasn't 
useful to send that non-subscriber's comments to the list anyway).

Most of the time the verification would fail in the MAIL FROM response, 
but occasionally it would fail in RCPT TO response.  So I ended up 
having to send MAIL FROM:<[email-masked]> or some such instead in 
order to get an accurate verification.

(one could argue that you shoudn't accept mail from misconfigured 
systems anyway, but bouncing such messages doesn't do anything useful 
IMHO because it doesn't report an error that is likely to be understood 
by the person receiving the bounced mail.  That and you have to arrange 
for the bounces to have a return-path other than <> in order for them 
to even get back to the sender.)

>> If the CBV doesn't send MAIL FROM:<>, what happens when the SMTP 
>> server
>> associated with the return-path server also uses a CBV?
>
> Then the second system will perform a CBV as well.   An overhead or
> redundancy issue that should be address with the SMTP protocol.

Here's what I see happening if this is widely adopted:

A0->B0: MAIL FROM:<my.address>
-B0->A1: HELO
-B0->A1: MAIL FROM:<dummy.address@B>
--A1->B1: HELO
--A1->B1: MAIL FROM:<dummy.address@A>
---B1->A2: HELO
---B1->A2:MAIL FROM:<dummy.address@B>
----A2->B2: HELO
----A2->B2: MAIL FROM:<dummy.address@A>
...

You get the idea.  Note that the various An don't all have to be the 
same hosts; they're just hosts acting on behalf of domain A (A1..An are 
MXes for A).  Similarly for B0..Bn.  So it's not necessarily easy for 
either of these domains to "break the loop" by realizing that they're 
being asked to verify a dummy return address that is used to verify a 
real address.

> I think the more appropriate question is:
>
> What is the RFC 2821 presumption for the validity of the return path?
>
> Or at what point should RFC 2821 presume the return path to be valid?  
> When
> it is provided or when its too late (after the mail is received and
> rejected)?

Return-path is intended for (non)delivery notification, not for sender 
verification.  It is not reasonable to take return-path as an indicator 
of sender identity.

SMTP servers _should_ return immediate error conditions (in response to 
MAIL FROM, RCPT TO, DATA) whenever possible because, for a wide variety 
of reasons, immediate reporting is much more reliable and consumes less 
resources than bouncing the message.  For similar reasons, SMTP senders 
_should_ minimize the relay path length to the receiver's MX rather 
than aggregating traffic through upstream relays.  Of course there are 
reasons to have both outbound and inbound relays, but these do come at 
a cost.

Despite my use of CBV on mailing list traffic, I don't think they are 
nearly as good an indicator for traffic in general, for lots of 
reasons:

- The presumption that traffic with an invalid return-path is invalid, 
while perhaps reasonable for some mailing lists, is less reasonable as 
a general rule.
- The verification has lots of overhead, and some potential for looping 
if widely adopted.
- The effectiveness of CBV has decreased considerably over time.  When 
I first started using it (not sure when, maybe 7 years ago) it 
identified nearly 100% of spam sent to my lists.  When I stopped using 
it on a large scale (just over a year ago) it only identified about 25% 
of the spam sent to my lists.  It appeared that spammers had realized 
that mail from invalid addresses was less successful, so they simply 
started sending mail from valid addresses.
- In my experience, CBV does not provide 100% true negatives - because 
of misconfigured systems that reject MAIL FROM:<> and other valid mail. 
  (if you have your SMTP server report 4xx whenever the CBV tempfails, 
there are other problems - it is not unusual for a CBV to fail because 
either the sender's DNS servers or the sender's SMTP server are too 
slow or inaccessible, even though the sender's system is sending 
outgoing mail.)

In short - CBV is expensive for the receiver and easily defeated by 
spammers.  If widely adopted it very quickly becomes useless overhead.

I do think we need reliable traceability to the sender, but doing CBV 
on return-path is not the way to get it.

> The way I view it is the following:
>
> 1) The CAN-SPAM Act has provided a mandate with two SMTP level
> "requirements:"
>
> - Valid Return Paths
> - Advertisement Tags
>
> The marketing of CAN-SPAM Ready Systems (on both sides) has already 
> begun.
> At a minimum, this will place product requirements to provide auditing 
> and
> tracking.

CAN-SPAM is a crock.  We should not presume that the US Congress is 
qualified to dictate technical standards - heck, they can't even write 
nontechnical laws that are in the public interest.

> 2) Mr. Crocker's ASRG Proposal Guideline Document,
> draft-crocker-spam-techconsider-02.txt, emphasizes incremental and 
> backward
> compatibility minimizes or quelling any desire to fundamentally alter 
> the
> SMTP protocol.

I haven't read this document yet, so I can't comment on the extent to 
which I think the argument in the document is valid.  But using 
return-path as a sender identity conflicts with its purpose in SMTP as 
the destination of (non)delivery reports.  The Sender field from RFC 
822 is the closest thing to a verifiable sender identity from the 
original design, but it is so widely misused today that it is not 
salvageable. Basically the requirements for verification are such that 
we're going to need a new field.

Keith

Subject: Re: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
From: "B. Johannessen" <[email-masked]>
To: [email-masked]
Organization: http://db.org/
Date: Wed, 14 Jan 2004 09:54:16 +0000
Sender: [email-masked]

On Tue, 2004-01-13 at 21:59, Hector Santos wrote:
> Like I said, I thought about pleading the 5th here because I understand very
> clearly the sensitive nature of this topic.  I have much to say, yet it may
> not be all worth it the stress that it may generate (especially for those
> who are philosophically oppose, right or wrong).

I for one found this message most informative. And I'm leaning
towards agreeing with those who are "philosophically opposed".

More information on practical use of SMTP call-backs would be
a good thing for both sides of the debate. It can only increase
the quality of the discussion.

> I can provide more detail information if you like.

I say go ahead.

	Bob

-- 
-=[ B. Johannessen | [email-masked] -=- http://db.org/ | +4797152009 ]=-
-=[ Mail & Spam - News, Drafts & Standards - http://db.org/blog/ ]=-
-=[ On the Origin Of Spam: Spam Statistics - http://db.org/spam/ ]=-

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: CBV systems - was Re: SMTP Extensions - proper peply code for disabled commands
Date: Tue, 13 Jan 2004 16:59:29 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Tuesday, January 13, 2004 12:22 PM
Subject: Re: SMTP Extensions - proper peply code for disabled commands

> I assume that this CBV service uses SMTP to determine whether the
> return-path is valid.

Your presumption is correct.

hmmmmmm,  Maybe I should plead the 5th?  :-)

> If so, does that CBV send MAIL FROM:<>?

Configurable.  However, a NULL address is highly suggested for the reason
you suspect.

> If so, what happens when the SMTP server associated with the return-path
(incorrectly) returns an
> error to that? (in my experience this is a fairly common bug, or
> misfeature, or misconfiguration).

In our testing with over 25 commerical systems, millions of transactions.  I
believe I have personally seen logs with maybe a few of these type of
sessions. In all cases, it was confirmed to be "zombie-like" sites that
intentionally fail at the protocol level.

In our system,  a presumption of a non-compliant remote system is made with
a less than 100% trust value provided to the overall result.   Since I have
a pretty good idea where you are going with this, I should note a common
non-understanding of CBV systems (atleast for our system) is that it focuses
on looking for 100% true negatives (return paths rejected).  Little or no
trust is provided to those systems that accept the return path, hence NO
decision is made.    However, in our system,  SMTP compliancy it inherently
part the total process.   I made it a SYSTEM POLICY issue that the system
admin can choose.   Emperically, we have no found any problems (false
negatives) and when there were, their filter tables were quickly adjusted.

The bottom line is that it works in testing verifible BAD addresses.  Not
GOOD addresses because no trust can be put on this type of result.  This is
a KEY focus I wish to make to any one who may be philosophically against a
CBV.  I understand very clearly be a heated debate, especially for many who
have their mind already made up.  But most of the arguments I have seen is
that you can trust an accepted return path.  That is true.   The trust is
the rejected return path.

Other issues of OVERHEAD and REDUNDANCY was left to the optimization phase
of the project. They were also addressed.

> If the CBV doesn't send MAIL FROM:<>, what happens when the SMTP server
> associated with the return-path server also uses a CBV?

Then the second system will perform a CBV as well.   An overhead or
redundancy issue that should be address with the SMTP protocol.

I think the more appropriate question is:

What is the RFC 2821 presumption for the validity of the return path?

Or at what point should RFC 2821 presume the return path to be valid?  When
it is provided or when its too late (after the mail is received and
rejected)?

These are some of the things I would like to discuss in the near future when
I finish collecting my thoughts in how to best address or discuss the issues
regarding anonymous access exploitation or abuse of the SMTP protocol with
pragmatic gurus of the industry.   A realistic problem that will need to be
addressed sooner than later.

Like I said, I thought about pleading the 5th here because I understand very
clearly the sensitive nature of this topic.  I have much to say, yet it may
not be all worth it the stress that it may generate (especially for those
who are philosophically oppose, right or wrong).   I will leave you with
this in the hope it summarizes most of my thoughts:

1) Anonymous access abuse is the issue, not authorized or authenticated
access.

2) Both theoretically and empirically, the CBV has proved to work to provide
100% true negatives (return path rejected at the remote system).  100% trust
can be put behind this result.

3) Less than 100% trust can be provided to positive results (return path
accepted at the remote system).

4) Implementations can impose other test to measure the trust value of the
positive result.  In our case, we offer a open relay test option.  "Test to
see if ANY obfuscated non-local return path is also accepted."

5) Overhead and redundancy can be reduced a number of ways.  Primarily,
minimize the calls to when they are 100% necessary.  Domain/IP pairs called
DIPs are used as part of an automatic learning process.   The most important
overhead reduction is to call CBV iff a valid local recipient is provided.
In other words, allow the client to reach the RCPT TO state.  If your system
is going to refuse the RCPT TO address, then they is no need to call CBV.
We found that nearly 60% of the mail transaction is refused due to
non-existing recipients.  Hence that is a 60% CBV call reduction in a given
day.

6)  We also use LMAP-based DNS lookup proposals, specifically the DMP
proposal to help validate an authorized sender machine prior to using the
CBV.  This proved to be one of the troubling aspects I had with the ASRG key
focus of using DNS lookup solutions to address the anonymous access problem.
LMAP-based solutions only work for local domain validation.  It can not be
used to validate non-participating remote systems due to high DNS lookup
delays with spoofed domains.

7) Finally, empirical results with over 25 sites processing millions of
messages have shown overwhelming high success.

I can provide more detail information if you like.

Our system also works as a web service.  If you would to test it for
yourself, click the following URL. I will temporary open access to the web
service for the next few days:

    http://www.winserver.com/testwcsap

Statistics can be found at:

    http://www.winserver.com/public/Security

The way I view it is the following:

1) The CAN-SPAM Act has provided a mandate with two SMTP level
"requirements:"

- Valid Return Paths
- Advertisement Tags

The marketing of CAN-SPAM Ready Systems (on both sides) has already begun.
At a minimum, this will place product requirements to provide auditing and
tracking.

2) Mr. Crocker's ASRG Proposal Guideline Document,
draft-crocker-spam-techconsider-02.txt, emphasizes incremental and backward
compatibility minimizes or quelling any desire to fundamentally alter the
SMTP protocol.

I agree.

Hence, in other to satisfy both requirements, the only logical conclusion is
the validation of the return path using the current SMTP standard protocol.
However, the advertisement tag requirement presents a data collection
requirement with the standard protocol. This conflicts with dynamic return
path validation unless it is done at the delayed to the data collection
level as well.

In short, there are some fundamental conflicts with the RFC 2821
specification, the SMTP protocol and the mandate imposed by the CAN-SPAM
act.  Adjustments or rather "clarifications" need to be made in the RFC.

How that is done is the matter left up to discussion.

In the mean time,  our primary or main focus was to re-engineer our software
to provide programmable hooks at each state of the protocol state machine.
See the diagram at http://www.winserver.com/public/antispam

We choose to use  CBV concept as a validation sequence at the RCPT TO simply
because there is NO other way to do it.  We optimized it using intelligent
learning/caching and also implemented RBL and the ASRG DMP DNS Lookup
proposal to reduce the need to call the CBV.

What remains in our view is:

1) Removing SMTP from the fuzzy interpretation of mail content, focusing on
access control,
2) How can two current SMTP systems best communicate to perform this
process,
3) Clarification of the RFC 2821 validity of the return path and WHEN it is
valid.

Maybe I should of taken the 5th?  <g>

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Tue, 13 Jan 2004 12:22:30 -0500
From: Keith Moore <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Sender: [email-masked]

> Our server is calling a CBV (Callback Verifier) service, apparently
> the "+" signed is being escaped incorrectly.

I assume that this CBV service uses SMTP to determine whether the
return-path is valid.  

If so, does that CBV send MAIL FROM:<>?  If so, what happens when the
SMTP server associated with the return-path (incorrectly) returns an
error to that? (in my experience this is a fairly common bug, or
misfeature, or misconfiguration).  

If the CBV doesn't send MAIL FROM:<>, what happens when the SMTP server
associated with the return-path server also uses a CBV?

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Date: Tue, 13 Jan 2004 02:04:32 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Hector Santos" <[email-masked]>; "IETF-SMTP"
<[email-masked]>
Sent: Tuesday, January 13, 2004 12:50 AM
Subject: Re: SMTP Extensions - proper peply code for disabled commands

> p.s. Your server seems to not like Reverse-paths that contain
> plus signs, or otherwise has problems "verifying addresses" with
> which other systems seem to have no difficulty.    Might be
> worth trying to fix if you expect to participate in IETF
> discussions, since a number of folks around here use addresses
> of that type.
>

Our server is calling a CBV (Callback Verifier) service, apparently the "+"
signed is being escaped incorrectly.

I'll take care it pronto, in fact, its fixed now.

Thanks for pointing it out.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Tue, 13 Jan 2004 00:50:21 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, IETF-SMTP <[email-masked]>
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Sender: [email-masked]

Just to close this loop, the omission of 502 from the valid code 
list for EHLO was intentional.  Specifically, Lyndon is correct: 
2821 requires ("MUST") that EHLO be supported.  If you are not 
going to support it, then you are either operating in 821-land, 
in which EHLO is an unrecognized command, or you need the reply 
code for "my implementation is broken".

As Lyndon says, 2821 doesn't specify the details of ways to 
violate the standard. Consequently, it hence doesn't specify a 
value for the "my implementation is broken" code.  If you need 
one, I'd suggest, e.g., "999", which would cause almost all 
clients to stop talking to your server.  :-(

  regards,
    john

p.s. Your server seems to not like Reverse-paths that contain 
plus signs, or otherwise has problems "verifying addresses" with 
which other systems seem to have no difficulty.    Might be 
worth trying to fix if you expect to participate in IETF 
discussions, since a number of folks around here use addresses 
of that type.

--On Sunday, 11 January, 2004 00:20 -0500 Hector Santos 
<[email-masked]> wrote:

>
> Hello all,
>
> I am just throwing this out there for comment.
>
> - Proper response code for a disabled EHLO implementation.
>
> What is the proper response code for a system which supports
> EHLO but for some reason or another the sysop has it turned
> off (disabled)?
>
> In older code (versions back in 2001) we used a general 500
> response for commands that were not recognized or implemented
> but "turned off."
>
> In our current version, for recognized but turned off
> (disabled) commands the response was changed to use a 502
> response.   This was done in accordance to '2821' section
> 4.2.4:
>
> 4.2.4 Reply Code 502
>
>    Questions have been raised as to when reply code 502
> (Command not    implemented) SHOULD be returned in preference
> to other codes.  502    SHOULD be used when the command is
> actually recognized by the SMTP    server, but not
> implemented.  If the command is not recognized, code    500
> SHOULD be returned.  Extended SMTP systems MUST NOT list
> capabilities in response to EHLO for which they will return
> 502 (or    500) replies.
>
> However,  per 2821, it you look at the possible error codes
> for EHLO, it indicates 504, 550 as the possible error response
> codes.
>
> 504 says "command parameters are incorrect"
> 550 ... command rejected for policy reasons
>
> We didn't use 504 is because this seems to suggest the command
> is "possible" with proper parameters.
>
> We didn't use 550 because it sounds more like a "access issue"
> not a disabled notification. Also since section 4.2.4  seems
> to reinforce  502 as the proper response for a "recognized but
> not implemented" command, it may sense to use 502.
>
> What say you here?
>
> First,  in my view,  I think the document should clarify or
> distinguish the difference between  the terms;
>
>     - recognized,
>     - implemented and
>     - disabled.
>
> Second,  to me, it sounds like the specific responses can be
> summarized as:
>
>     500 is used for an unrecognized command
>     502 is used for recognized but disabled commands
>     504 is used for recognized but with incorrect parameters.
>     55x is used as a processed command but rejected for some
> reason
>
> would you agree with this?
>
> Anyway,  it seems obvious to understand that per section
> 4.2.4,  502 is inherently inclusive of all possible error
> responses for all ESMTP extended commands which are recognized
> but not implemented (to me, that means disabled).
>
> However, in section 4.3.2,  502 is explicitly stated as
> possible error codes for the commands such as VRFY, EXPN.
> Therefore, at a minimum,  502 should be added as a possible
> error code for the EHLO command listed in section 4.3.2.
>
> Do you agree?
>
> Thanks

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: Submitting new SMTP related RFC drafts 
Date: Mon, 12 Jan 2004 23:46:22 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Monday, January 12, 2004 11:11 PM
Subject: Re: Submitting new SMTP related RFC drafts

On Mon, 12 Jan 2004 18:02:30 EST, Hector Santos
<[email-masked]>  said:

>> Is this mailing list a good place to bring up and discuss SMTP
>> ideas/extensions?

> As good a place as any.....

Sounds like a good title for a new Jack Nicholson movie... :-)

Thanks John, Valdis for your excellent input.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 12 Jan 2004 23:42:15 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], Hector Santos <[email-masked]>
cc: IETF-SMTP <[email-masked]>
Subject: Re: Submitting new SMTP related RFC drafts
Sender: [email-masked]

Hector,

I would add one thing to Valdis's helpful observations below...

The text in the last two paragraphs of section 2.2.1 of RFC 2821:

	SMTP's strength comes primarily from its simplicity.
	Experience with many protocols has shown that protocols
	with few options tend towards ubiquity, whereas
	protocols with many options tend towards obscurity.
	
	Each and every extension, regardless of its benefits,
	must be carefully scrutinized with respect to its
	implementation, deployment, and interoperability costs.
	In many cases, the cost of extending the SMTP service
	will likely outweigh the benefit.

Is usually taken quite seriously.   So you should not be 
surprised if some of the responses to an otherwise-good 
suggestion take the form of "that might help you, but most of 
the Internet doesn't need it, so it isn't worth the clutter in 
the protocol".

regards,
    john

--On Monday, 12 January, 2004 23:11 -0500 
[email-masked] wrote:

> On Mon, 12 Jan 2004 18:02:30 EST, Hector Santos
> <[email-masked]>  said:
>
>> Is this mailing list a good place to bring up and discuss SMTP
>> ideas/extensions?
>
> As good a place as any..
>
>> How about the current RFC 2821 document.  I have some
>> concerns and comments to make about it.  I don't wish to
>> ruffle any feathers, make enemies  and certainly, I don't
>> wish to go over stuff already covered or being handled by
>> some group or person, etc.  I would like to do this the
>> "right way."
>
> A good way to do this is to ask "Has anybody ever thought
> about doing XYZ?". Quite likely, either somebody has tried it,
> or thought about it.  If not, we'll be happy to at least look
> at the idea.
>
> If you have an idea for an ESMTP protocol extension, feel free
> to bring it up. Don't bother trying to do an entire spec
> first, that's usually a bad idea.  What you want to start with
> are:
>
> 1) A problem statement: "I could tune my SMTP software to be
> much more efficient if the other end told me XYZ up front.
> Not knowing it means I have to do A, but if I knew XYZ, I
> could optimize by doing B".
>
> 2) A *approach* to the problem - do you need to know XYZ
> per-recipient, or per-connection, or would it make sense to
> cache it per-site or per-domain?
>
> 3) Spend at least a little time thinking about obvious
> problems - "XYZ only makes sense within a domain", "You'd need
> prior arrangements to exchange data before using XYZ", "A
> rogue client/server could horque things up by lying about
> XYZ", and so on...
>
> 4) Be prepared for feedback, and be ready to incorporate it if
> it makes sense. I can guarantee you that you'll possibly get
> anything from "http would be a better way to attack that" to
> "that won't work because there's a requirement ABC" to "if
> you're going to do that, think about this too"...
>
> And if you get flamed, just remember - there's people on the
> list who are generally listened to because they've been around
> and are generally wise and worth listening to.  But I've said
> dumb things here and been told so, and even having written
> RFC822 won't save you from being told that you've come up with
> a bad idea, so don't take it personally. :)
>

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: Submitting new SMTP related RFC drafts 
From: [email-masked]
Date: Mon, 12 Jan 2004 23:11:26 -0500
Sender: [email-masked]

--==_Exmh_-968764388P
Content-Type: text/plain; charset=us-ascii

On Mon, 12 Jan 2004 18:02:30 EST, Hector Santos <[email-masked]>  said:

> Is this mailing list a good place to bring up and discuss SMTP
> ideas/extensions?

As good a place as any..

> How about the current RFC 2821 document.  I have some concerns and comments
> to make about it.  I don't wish to ruffle any feathers, make enemies  and
> certainly, I don't wish to go over stuff already covered or being handled by
> some group or person, etc.  I would like to do this the "right way."

A good way to do this is to ask "Has anybody ever thought about doing XYZ?".
Quite likely, either somebody has tried it, or thought about it.  If not, we'll be
happy to at least look at the idea.

If you have an idea for an ESMTP protocol extension, feel free to bring it up.
Don't bother trying to do an entire spec first, that's usually a bad idea.  What you
want to start with are:

1) A problem statement: "I could tune my SMTP software to be much more efficient
if the other end told me XYZ up front.  Not knowing it means I have to do A, but
if I knew XYZ, I could optimize by doing B".

2) A *approach* to the problem - do you need to know XYZ per-recipient, or
per-connection, or would it make sense to cache it per-site or per-domain?

3) Spend at least a little time thinking about obvious problems - "XYZ only
makes sense within a domain", "You'd need prior arrangements to exchange data
before using XYZ", "A rogue client/server could horque things up by lying about
XYZ", and so on...

4) Be prepared for feedback, and be ready to incorporate it if it makes sense.
I can guarantee you that you'll possibly get anything from "http would be a
better way to attack that" to "that won't work because there's a requirement
ABC" to "if you're going to do that, think about this too"...

And if you get flamed, just remember - there's people on the list who are
generally listened to because they've been around and are generally wise and
worth listening to.  But I've said dumb things here and been told so, and even
having written RFC822 won't save you from being told that you've come up with a
bad idea, so don't take it personally. :)

--==_Exmh_-968764388P
Content-Type: application/pgp-signature

--==_Exmh_-968764388P--

Date: Mon, 12 Jan 2004 21:20:10 -0500
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, IETF-SMTP <[email-masked]>
Subject: Re: Submitting new SMTP related RFC drafts
Sender: [email-masked]

--On Monday, 12 January, 2004 18:02 -0500 Hector Santos 
<[email-masked]> wrote:

>...
> How about the current RFC 2821 document.  I have some concerns
> and comments to make about it.  I don't wish to ruffle any
> feathers, make enemies  and certainly, I don't wish to go over
> stuff already covered or being handled by some group or
> person, etc.  I would like to do this the "right way."

Suggestions fall into two categories...

(1) Substantive suggestions that actually impact how the 
protocol works.  I.e., 2821 specifies that things should work in 
one way, you think they should work differently.

	Those suggestions should be sent to this list.

(2) Suggestions that do not have any protocol impact, but that 
are about textual errors or areas where the language could be 
made more clear or easy to understand.

	Send those suggestions directly to me, preferably at
	[email-masked]

If you are in doubt as to whether a suggestion fits in the first 
or second categories, it is _much_ better to send it to the list.

regards,
        john

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Submitting new SMTP related RFC drafts
Date: Mon, 12 Jan 2004 18:02:30 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I am new to this IETF and RFC drafting process, so please excuse any naivety
I may show.

I believe I downloaded all the material I need (how to rfcs, templates, etc)
for creating RFC drafts.

Questions:

Is this mailing list a good place to bring up and discuss SMTP
ideas/extensions?

1id-guidelines.txt indicates I should send new drafts to
[email-masked] and at the same time cc: to the chair(s) of the
"working group."    Is there a working group for IETF-SMTP?  or this list
the working group?

How about the current RFC 2821 document.  I have some concerns and comments
to make about it.  I don't wish to ruffle any feathers, make enemies  and
certainly, I don't wish to go over stuff already covered or being handled by
some group or person, etc.  I would like to do this the "right way."

Any additional suggestions based on your experience would be appreciated.

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "Lyndon Nerenberg" <[email-masked]>, "IETF-SMTP" <[email-masked]>
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Date: Sun, 11 Jan 2004 22:17:15 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Arnt and Lyndon

You are both right.  Thanks for your input on this.    I guess the engineer
who implemented Extended SMTP features needed to make it an OPTION since it
was something "new" for customers.  So using the 500 response code was
correct in this regard.  I screwed it up by changing it to 502.

Anyway thanks for both your input.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Lyndon Nerenberg" <[email-masked]>
To: "Hector Santos" <[email-masked]>; "IETF-SMTP"
<[email-masked]>
Sent: Sunday, January 11, 2004 2:39 PM
Subject: Re: SMTP Extensions - proper peply code for disabled commands

>
>
> > What is the proper response code for a system which supports EHLO but
> > for some reason or another the sysop has it turned off (disabled)?
>
> If you don't support EHLO then 2821 no longer applies, and so you have
> to follow 821 where EHLO is an unrecognized command, thus the correct
> response is 500.
>
> But why disable EHLO? If there are specific services you don't want to
> support, don't advertise them in the EHLO response. By accepting the
> EHLO you'll avoid the extra round-trips incurred by the fallback to
> HELO.
>
> --lyndon
>

Date: Sun, 11 Jan 2004 12:39:03 -0700
From: Lyndon Nerenberg <[email-masked]>
To: Hector Santos <[email-masked]>, IETF-SMTP <[email-masked]>
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Sender: [email-masked]

> What is the proper response code for a system which supports EHLO but
> for some reason or another the sysop has it turned off (disabled)?

If you don't support EHLO then 2821 no longer applies, and so you have 
to follow 821 where EHLO is an unrecognized command, thus the correct 
response is 500.

But why disable EHLO? If there are specific services you don't want to 
support, don't advertise them in the EHLO response. By accepting the 
EHLO you'll avoid the extra round-trips incurred by the fallback to 
HELO.

--lyndon

From: Arnt Gulbrandsen <[email-masked]>
To: Hector Santos <[email-masked]>
Subject: Re: SMTP Extensions - proper peply code for disabled commands
Cc: [email-masked]
Date: Sun, 11 Jan 2004 16:33:43 +0100
Sender: [email-masked]

According to RFC 2821 section 4.5.1, EHLO MUST be supported. You cannot 
disable that command and remain conformant.

Of course RFC 2821 doesn't specify how to violate RFC 2821.

In the general case, you can disable commands by not announcing them in 
the EHLO response. If the implementation supports ETRN but the admin 
disables it, you simply leave ETRN out of the EHLO response.

Arnt

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: SMTP Extensions - proper peply code for disabled commands
Date: Sun, 11 Jan 2004 00:20:34 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Hello all,

I am just throwing this out there for comment.

- Proper response code for a disabled EHLO implementation.

What is the proper response code for a system which supports EHLO but for
some reason or another the sysop has it turned off (disabled)?

In older code (versions back in 2001) we used a general 500 response for
commands that were not recognized or implemented but "turned off."

In our current version, for recognized but turned off (disabled) commands
the response was changed to use a 502 response.   This was done in
accordance to '2821' section 4.2.4:

4.2.4 Reply Code 502

   Questions have been raised as to when reply code 502 (Command not
   implemented) SHOULD be returned in preference to other codes.  502
   SHOULD be used when the command is actually recognized by the SMTP
   server, but not implemented.  If the command is not recognized, code
   500 SHOULD be returned.  Extended SMTP systems MUST NOT list
   capabilities in response to EHLO for which they will return 502 (or
   500) replies.

However,  per 2821, it you look at the possible error codes for EHLO, it
indicates 504, 550 as the possible error response codes.

504 says "command parameters are incorrect"
550 ... command rejected for policy reasons

We didn't use 504 is because this seems to suggest the command is "possible"
with proper parameters.

We didn't use 550 because it sounds more like a "access issue" not a
disabled notification. Also since section 4.2.4  seems to reinforce  502 as
the proper response for a "recognized but not implemented" command, it may
sense to use 502.

What say you here?

First,  in my view,  I think the document should clarify or distinguish the
difference between  the terms;

    - recognized,
    - implemented and
    - disabled.

Second,  to me, it sounds like the specific responses can be summarized as:

    500 is used for an unrecognized command
    502 is used for recognized but disabled commands
    504 is used for recognized but with incorrect parameters.
    55x is used as a processed command but rejected for some reason

would you agree with this?

Anyway,  it seems obvious to understand that per section 4.2.4,  502 is
inherently inclusive of all possible error responses for all ESMTP extended
commands which are recognized but not implemented (to me, that means
disabled).

However, in section 4.3.2,  502 is explicitly stated as possible error codes
for the commands such as VRFY, EXPN. Therefore, at a minimum,  502 should be
added as a possible error code for the EHLO command listed in section 4.3.2.

Do you agree?

Thanks

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Thu, 08 Jan 2004 00:51:50 -0600
From: "Eric A. Hall" <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>
CC: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/7/2004 12:09 PM, Richard O. Hammer wrote:

> I would argue that a new reply code be assigned to SMTP server timeout.
> 
> I would argue against using 421 because that means something 
> distinctly different.  A well programmed SMTP client which notices 
> that it has received a 421 may conclude "Service not available" 
> (reference Sections 4.2.2, 4.2.3), may therefore add the host to "a 
> list of hosts it cannot reach" (reference Section 4.5.4.1), and may 
> erroneously postpone further attempts to reach that host.

I read 4.5.4.1 to mean that the message should be requed and the host
should be retried since a transient error occurred during the session, not
a reachability error that prevented the session from being established.

Using an enhanced status code could be useful, but that does not need to
be specified in 2821bis. ESN codes 4.4.1 and 4.4.2 both seem applicable,
although a more-specific ESN would probably be more useful. A separate
document describing such a code would also serve as a useful vehicle for
explicitly describing the appropriate behavior. So, it may be that a
minimal clarification to 2821 can get us most of the way there and a
separate document can get us the whole way.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 08 Jan 2004 00:41:54 -0600
From: "Eric A. Hall" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: John C Klensin <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/7/2004 12:05 PM, Hector Santos wrote:

> hmmmmm, lets see, you send an unsolicited 421 notification, delay the
> hangup for 1 minute to see if the client will do a graceful exit
> (quit)?   Is that what we looking for here?   Lets assume Current
> software will "wake up", what happens next depends on whether it purses
> the notification or not.  If not,  the client will exit on its own.

If someone wants a signal for "about to hangup" with a pause in case the
client resumes, then that would best be served with a 32x code. While that
may be useful, it would also be a fairly significant expansion, and I
don't think that's reasonable for the scope of this discussion.
Conversely, a statement that unilateral disconnects should be preceded
with a 421 is a clarification to existing functionality I think.

> I'm just trying to see where  the "idea" would apply.  Waiting on a
> QUIT to complete a transaction?

421 means that the server is going away, so anything more from the client
would have to be handled with a new session.

> I am just saying that we shouldn't be expecting clients to be
> "kept-alive." with "wake-up warnings."  as this will require a
> completely new framework.

I don't think that's an appropriate objective here either.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 07 Jan 2004 13:09:17 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

I would argue that a new reply code be assigned to SMTP server timeout.

I would argue against using 421 because that means something 
distinctly different.  A well programmed SMTP client which notices 
that it has received a 421 may conclude "Service not available" 
(reference Sections 4.2.2, 4.2.3), may therefore add the host to "a 
list of hosts it cannot reach" (reference Section 4.5.4.1), and may 
erroneously postpone further attempts to reach that host.

Also, although John is probably correct in assuming that the client 
has already dropped the connection when the server reaches timeout, in 
the large majority of cases, I guess there will inevitably be a few 
cases when the client does come back and read what the server has last 
written.  In these few cases a distinct code for timeout could clarify 
what happened.

In one more case I can imagine good coming from a distinct timeout 
code.  What if the timeout is malfunctioning? happening too quickly? 
The administrator at the client end could find information in his logs 
which might help the administrator at the server end debug the problem.

However, I accept there must be very good reasons for not expanding 
the list of reply codes.  Further, as experience produces more 
encounters with antagonistic SMTP clients, it may be best to keep your 
cards close to your chest; in a hostile environment it can be a 
mistake to give more information.  So I cannot judge entirely whether 
the benefits of adopting a distinct reply code for timeout would 
outweigh the costs.

Nonetheless I will respond to John's request for a suggestion for what 
the new text might say.

FIRST
Add a third "not...except" point to Section 3.9:

- After waiting too long for the next command from the client or after 
detecting too slow a rate of data transmission.  In this case the 
server may either close the connection silently or send a 422 response 
code before closing the connection.  See section 4.5.3.2.

SECOND
In Section 4.5.3.2, after the last sentence (which says "An SMTP 
server SHOULD have a timeout of at least 5 minutes while it is 
awaiting the next command from the sender.") add:

It may [or SHOULD?] also monitor the rate transmission while it is 
receiving the message data and may [or SHOULD?] time out if this rate 
is too slow.

THIRD
the new code would need to be added into the lists in both Sections 
4.2.2 and 4.2.3, such as:

422 Timeout: either too much delay between commands or too slow a rate 
of data transmission.

I cannot defend the particular number, 422, as being better than other 
4xx, or even possibly 5xx, reply codes for this timeout situation.  I 
picked 422 mostly because it is so close to 421, with which it shares 
the aspect of asynchronous termination by the server.

My suggestions above are tentative, of course.  RFC 2821 is already a 
large and complex document, in which a given subject (such as the 
current subject of timeouts) may be addressed in several sections, and 
great care (which I have not exercised in the above, tentative, 
suggestions) must be needed to change one section without breaking the 
intent of another section.  You have my sympathy, John.

Rich Hammer
Hillsborough, N.C.
mailscreen.net

From: "Hector Santos" <[email-masked]>
To: "John C Klensin" <[email-masked]>, "Paul Smith" <[email-masked]>, <[email-masked]>
Subject: Re: what to say on timeout?
Date: Wed, 7 Jan 2004 13:05:26 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Paul Smith" <[email-masked]>; <[email-masked]>
Sent: Wednesday, January 07, 2004 10:10 AM
Subject: Re: what to say on timeout?

>
> Sigh.
>
> Folks, I'm happy to rework the text if there is consensus that
> change is actually needed, and, ideally, if someone supplies the
> suggested text.  However, after rereading the text, and the
> predecessor text in RFC 821, it actually seems clear to me what
> is intended...

Hi John,

I believe so as well.

In my view, the SMTP client has the ultimate design responsibility to decide
whether it needs to retry a transaction.   Lets clarify if there is any
concern here it would be a command mode idle timeout.  Request processing
timeouts is an non-issue. A 421 response is appropriate with a hangup.   The
client will react accordingly when aborted.

For command mode timeouts, the client will react accordingly as well with
either non-passive or passive disconnects.

However, in my view, it is a engineering mistake to open the door to
"unnatural" and "unexpected" designs by beginning to allow for non-passive
unsolicited server notifications for command-mode timeouts.

Will it be harmless?  Quite likely it will be, but I don't see why unless
you are introduction a "Wake up" Concept in which case, we need a new CODE
that identifies a "heart beat,"  "wake-up" or "NOOP" notification concept at
the application protocol level, not the TCP level that will minimize the
potential for a "still alive" client to use a 421 notification as part of
the next command.  Otherwise you defeat the purpose.  A new code is
required.

hmmmmm, lets see, you send an unsolicited 421 notification, delay the hangup
for 1 minute to see if the client will do a graceful exit (quit)?   Is that
what we looking for here?   Lets assume Current software will "wake up",
what happens next depends on whether it purses the notification or not.  If
not,  the client will exit on its own.

I'm just trying to see where  the "idea" would apply.  Waiting on a QUIT to
complete a transaction?  What completes a transaction?  I only see this when
he client completes the DATA, and receives a server notification but stills
idle now.

In my view, if we are going to have such a  "Keep-Alive design"  that it
should be 100% based on a SMTP client design and this can be done with no
changes to the specs:

a) SMTP client begins transaction with the MAIL FROM: command,

b) SMTP client issues the RSET to complete a transaction, and waits for new
mail.

c) Understanding per specification the server will have a minimum 5 minute
command mode idle timeout,  the client SHOULD send a NOOP command to the
server every minute.

d) The client eyeballing for new queued mail begin a new transaction, go to
A:

I am just saying that we shouldn't be expecting clients to be "kept-alive."
with "wake-up warnings."  as this will require a completely new framework.

Look at this way:

User agents such as Outlook with a NNTP client 100% expects passive timeout
disconnects by design.  Non-passive server notifications will break the
client as the notification will be queued for the next NNTP command
response.  Here also, there is no server base "NOOP" or "wakeup" server
notification code that will cause the client to wake up, purge the
unsolicited notification and do something.   Instead,  what I've seen some
NNTP clients issue an empty or some "redundant" command if they wanted to
keep a low timeout NNTP server alive.

If we allow for unsolicited server notifications, then in my view, we will
break more things than it solves or addresses.

Just my view on the matter.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Wed, 07 Jan 2004 11:51:29 -0600
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/7/2004 10:59 AM, [email-masked] wrote:

> On Wed, 07 Jan 2004 10:05:11 CST, "Eric A. Hall" said:
> 
>> The client could simply be waiting on some other event or input, like
>> a password for AUTH, and not borked or dead.
> 
> A case could be made that if you've been sitting at a password prompt
> for THAT long, the liveware has probably popped out to Applebee's for
> lunch and the connection is *effectively* borked.

...so they should be made to feel the wrath of valdis, and telling them
that the session has timed out would ease some of their discomfort, and we
can't have that now can we  :)

> Where do you draw the line? 

the 'line' is what the design principles say, which is that application
errors should be reported in the application stream.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked]
Subject: Re: what to say on timeout? 
From: [email-masked]
Date: Wed, 07 Jan 2004 11:59:11 -0500
Sender: [email-masked]

--==_Exmh_888411763P
Content-Type: text/plain; charset=us-ascii

On Wed, 07 Jan 2004 10:05:11 CST, "Eric A. Hall" said:

> The client could simply be waiting on some other event or input, like a
> password for AUTH, and not borked or dead.

A case could be made that if you've been sitting at a password prompt for THAT
long, the liveware has probably popped out to Applebee's for lunch and the
connection is *effectively* borked.

Where do you draw the line?  Somebody runs "my_mta -v | more" and it wedges
on flow control - what do you do there?

--==_Exmh_888411763P
Content-Type: application/pgp-signature

--==_Exmh_888411763P--

Date: Wed, 07 Jan 2004 16:46:17 +0000
To: John C Klensin <[email-masked]>, [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: what to say on timeout?
Sender: [email-masked]

>The two bulleted statements are inapplicable, not because they don't list 
>this case, but because the SMTP server isn't "intentionally closing the 
>connection".  Instead, it has detected a state it interprets as the client 
>having dropped the logical connection (even if the TCP connection is still 
>up).

Actually, I'd read a 'timeout connection close' as 'intentionally closing 
the connection'. If the server 'does nothing', then the connection could 
well stay open indefinitely. So the server has to close the connection. 
Hopefully, it'll do this deliberately, not by accident. So, in my reading 
it's an intentional close. (This isn't trying to be pedantic, just clear - 
if it's not clear, then there'll just need to be another SMTP standard 
after 2821 to clarify it..)

I don't think you can say that a timeout will only occur when the client 
has 'dropped the logical connection' (whatever that means). It's entirely 
possible the client is busy doing something else and is just taking a long 
time to do so (this is, presumably, why the timeout is limited to a minimum 
of 5 minutes, when, in reality, an SMTP client which is inactive for more 
than 30 seconds is probably 'dead' in most cases). So in this case it is 
definitely that the server is intentionally dropping the connection when 
the client is still happily connected.

So, I'd add 'timeout' to the list of possible causes of an intentional 
close of the connection.

>The next two paragraphs strengthen this impression -- they talk about 
>receipt of unknown commands and "shutdown via external means", which is 
>fairly clearly not the client timing out.

Hmm, I took that as meaning that the server shouldn't shut down the 
connection in any other situation (which clearly isn't what it was meant to 
mean, but seemed to me to be what it said).

>So, the timeout expires and the server concludes that the client is gone, 
>even though the TCP connection isn't.  What does the standard permit the 
>server to do then?  The answer --as is the answer several other places in 
>the SMTP specs-- is "anything sensible".

But that isn't what RFC2821 says.. It might be what it's meant to mean, but 
it doesn't say it.

>So, one way to interpret what the standard says today is
>
>         * There is _no_ specific and prescribed behavior because
>         there isn't intended to be any specific and prescribed
>         behavior.
>
>         * Servers are, as in all other things, supposed to
>         exercise good sense in the environment in which they
>         find themselves.
>
>Does that help?

Not really, as you say, that's "one way to interpret what the standard 
says"... What a standard should try to aim for is "the only way to 
interpret it", not "hopefully you'll interpret it the way I meant you to"

>Do we really need more text and, if so, what?

I'd say yes. If you want it to mean what you give as a possible way to 
interpret it - put that into the standard. I'd say if there's a point of 
uncertainty which isn't just one person being dense then it needs to be 
clarified. This one has a few people giving different responses, so, IMHO, 
it needs clarifying.

Personally, I'd say the 421 SHOULD be sent on any 'intentional closure' of 
the connection other than following a QUIT command. (This isn't actually 
what our own mail server does at the moment, so this isn't a vested interest)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Wed, 07 Jan 2004 10:05:11 -0600
From: "Eric A. Hall" <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/7/2004 9:10 AM, John C Klensin wrote:

> If a timeout expires, the assumption is that the mail session is 
> over, that the client has died, and that the server should take 
> an appropriate action.

The client could simply be waiting on some other event or input, like a
password for AUTH, and not borked or dead.

Separately, timeouts are server-based, and variable for each server, but
there is no mechanism for servers to inform clients of the timeouts in
use. It's entirely possible that a particulr server will timeout faster
than the client expects, although this is discouraged.

> (2) Is it sensible to send an asychronous 421 (or, under unusual 
> circumstances when one hopes to not hear about that message 
> again, a 5yz code) to the client, hoping that will wake up up, 
> wait a short interval to see if there is a response, and then 
> close the connection?
> 
> 	Yes, although, as others have commented on the list, it
> 	is unlikely to accomplish very much in most situations.
> 	On the other hand, it is, at worst, harmless.

> Do we really need more text and, if so, what?

The robustness principle requires us to assume that the client is not
borked but instead is merely waiting. The most conservative thing to do is
to tell the client why the session is being dropped unilaterally.

The end-to-end principle requires the application-layer to communicate
this kind of stuff, rather than relying on other layers.

SMTP requires explicit signalling in order for the queuing and routing
algorithims to work reliably. A 421 will force a requeue, and that is the
approriate recovery.

So, it seems to me that 421 is always appropriate and correct, regardless
of whether or not any other mechansisms (like TCP probes) are also tried.
As long as the socket isn't blocked, the server should send it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 07 Jan 2004 10:10:45 -0500
From: John C Klensin <[email-masked]>
To: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

Sigh.

Folks, I'm happy to rework the text if there is consensus that 
change is actually needed, and, ideally, if someone supplies the 
suggested text.  However, after rereading the text, and the 
predecessor text in RFC 821, it actually seems clear to me what 
is intended...

Please remember, in reading this, that 821 tried to define 
things independent of the particular properties of TCP and that 
2821, unless circumstances required something else, tried to 
preserve that model.

Given the synchronous model of SMTP, which several people have 
discussed on the list, the "timeout" limits are intended to let 
the server detect a client that has died or gone silent without 
closing the TCP connection.  SMTP's concern is that

	(i) this not be done too quickly, causing unnecessary
	mail delays, retries, and confusion (see the
	introduction to 4.5.3.2) and
	
	(ii) clients that have, in fact, died or disappeared do
	not tie up server or network resources indefinitely.

Now, all of the timeouts specified occur when the server is 
waiting for a command or data from the client.

If a timeout expires, the assumption is that the mail session is 
over, that the client has died, and that the server should take 
an appropriate action.   Now let's look at 3.9...

The two bulleted statements are inapplicable, not because they 
don't list this case, but because the SMTP server isn't 
"intentionally closing the connection".  Instead, it has 
detected a state it interprets as the client having dropped the 
logical connection (even if the TCP connection is still up). 
The next two paragraphs strengthen this impression -- they talk 
about receipt of unknown commands and "shutdown via external 
means", which is fairly clearly not the client timing out.

So, the timeout expires and the server concludes that the client 
is gone, even though the TCP connection isn't.  What does the 
standard permit the server to do then?  The answer --as is the 
answer several other places in the SMTP specs-- is "anything 
sensible".  "Sensible" may differ in different situations:

(1) Is it sensible to simply shut down the network-level 
connection?

	Yes.  By having the timeout expire, the server assumes
	that the applications-level connection is gone.

(2) Is it sensible to send an asychronous 421 (or, under unusual 
circumstances when one hopes to not hear about that message 
again, a 5yz code) to the client, hoping that will wake up up, 
wait a short interval to see if there is a response, and then 
close the connection?

	Yes, although, as others have commented on the list, it
	is unlikely to accomplish very much in most situations.
	On the other hand, it is, at worst, harmless.

(3) Is it sensible to send some sort of TCP signal, e.g., a SYN, 
to see if the client can be woken up.

	Yes, although, for lots of reasons, this shouldn't be
	"required" or even "strongly recommended" behavior, it
	certainly shouldn't be prohibited either.

So, one way to interpret what the standard says today is

	* There is _no_ specific and prescribed behavior because
	there isn't intended to be any specific and prescribed
	behavior.
	
	* Servers are, as in all other things, supposed to
	exercise good sense in the environment in which they
	find themselves.

Does that help?
Do we really need more text and, if so, what?

    john

--On Tuesday, 06 January, 2004 00:07 +0000 Paul Smith 
<[email-masked]> wrote:

>
> At 19:40 05/01/2004, you wrote:
>
>> Hector Santos wrote:
>>> Send a 421 per RFC 2821 section 3.9.
>
> I think RFC 2821 needs clarification here.
>
> Nowhere does it say what an SMTP server should do after a
> timeout. Section 3.9 says that the SMTP server MUST NOT close
> the connection except in two situations - which do not include
> a timeout situation...
>
> So, although section 4.5.3.2 allows (recommends) the SMTP
> server to have a timeout, it isn't allowed to close the
> connection after the timeout - so what should it do??
>
> I'm pretty sure this isn't what was intended by RFC 2821, but
> it's what it says..
>
> So, I'd suggest that 3.9 have an extra possible cause of a
> connection termination as a 'timeout situation as described in
> 4.5.3.2'. At this point in the RFC it could be suggested what
> the server should do.
>
> (The logical thing is probably to treat a timeout situation
> the same as a 'SMTP server shutdown situation' - ie it SHOULD
> attempt to send a line containing a 421 response code
> asynchronously)
>
>
> Paul				VPOP3 - Internet Email Server/Gateway
> [email-masked]			http://www.pscs.co.uk/
>
>

Date: 05 Jan 2004 22:29:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2821 Address Resolution
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Hector Santos)  wrote on 05.01.04 in <005d01c3d34a$38922b90$6401a8c0@FAMILY>:

> In regards to RFC2821, section 5, saying:

[...]

> There seems to be a conflict between this and the following itemized rule in
> RFC1123:

> What say you?

See at the beginning of 2821, where it says "Updates: 1123"?

MfG Kai

Date: Tue, 06 Jan 2004 00:07:09 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: what to say on timeout?
Sender: [email-masked]

At 19:40 05/01/2004, you wrote:

>Hector Santos wrote:
>>Send a 421 per RFC 2821 section 3.9.

I think RFC 2821 needs clarification here.

Nowhere does it say what an SMTP server should do after a timeout. Section 
3.9 says that the SMTP server MUST NOT close the connection except in two 
situations - which do not include a timeout situation...

So, although section 4.5.3.2 allows (recommends) the SMTP server to have a 
timeout, it isn't allowed to close the connection after the timeout - so 
what should it do??

I'm pretty sure this isn't what was intended by RFC 2821, but it's what it 
says..

So, I'd suggest that 3.9 have an extra possible cause of a connection 
termination as a 'timeout situation as described in 4.5.3.2'. At this point 
in the RFC it could be suggested what the server should do.

(The logical thing is probably to treat a timeout situation the same as a 
'SMTP server shutdown situation' - ie it SHOULD attempt to send a line 
containing a 421 response code asynchronously)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>, "Paul Smith" <[email-masked]>
Subject: Re: what to say on timeout?
Date: Mon, 5 Jan 2004 20:01:01 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Paul Smith" <[email-masked]>
To: <[email-masked]>
Sent: Monday, January 05, 2004 12:44 PM
Subject: Re: what to say on timeout?

>
> I don't think it really matters...
> If you send such a 'response' asynchronously, then either the client has
> died somehow and won't see the response, so it won't matter, or it won't
be
> listening for data and won't see the response so it won't matter, or it
> will see the response which will be useful.

ditto

> Actually, my reading of RFC 2821 shows an inconsistency....
>
> ...
>
> If it *IS* shutting down the SMTP service, (in which case 3.9's 'MUST NOT'
> is meaningless), then the rest of 3.9 says that you SHOULD send a 421
> response asynchronously, whether the client is expecting it or not.

I agree   However.  the way I finally "realized" the intent 3.9 in practice
was that it helps to minimize incomplete sessions that may incur unnecessary
retries by a sender.

    -  After receiving a QUIT command and responding with a 221 reply.

In my view, implies to allow the client to fulfill its transaction (accepted
mail or not) by allow it to drive the server to completion.

   -  After detecting the need to shut down the SMTP service and
      returning a 421 response code.  This response code can be issued
      after the server receives any command or, if necessary,
      asynchronously from command receipt (on the assumption that the
      client will receive it after the next command is issued).

In my view, implies that the possibility of passive disconnects may
perpetuate a retry, hence it is might be better to send "something."

However, I agree with you that in for a client idle state, disconnecting
with or without a 421 does not matter, especially since it is more than like
that the data will not be read by the client anyway due to the socket
disconnect.

From: "Hector Santos" <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>, <[email-masked]>
Subject: Re: what to say on timeout?
Date: Mon, 5 Jan 2004 19:37:45 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: <[email-masked]>
Sent: Monday, January 05, 2004 2:40 PM
Subject: Re: what to say on timeout?

> Can you say -- forthrightly with confidence -- what you imply? that
> there is no formalized or standard message for the SMTP server to send
> just before it exercises its option to timeout because of long
> inactivity or too slow data transmission?

Hi Richard,

Yes, because it doesn't make sense if you are going to hangup the socket
connection.  If you are not going to hangup, then it makes sense to send
something with the expecting the client might do something with it.

Two reasons:

First, I believe it is inherently technically understood that SMTP is an
uni-directional standard client/server conversational model with a fixed
state machine.  The design is simply not there for the server to send
unsolited server notifications that is not part of an client request.   But
you question this, right?

Second,  more importantly if you drop the connection, the client will detect
it and most likely ignore whatever might be in the receiver queue and close
its end of the socket.   Unless the client is using a TCP-half close socket
shutdown logic,  the data is lost.

Basically, in regards to SMTP,  unless I am completely unaware of some
"hidden" design,  in situations where there is a command idleness (not a
request processing state),  there is simply no SMTP client design that
allows for the SMTP client to react to unsolicated server notifications
unless you sent the unsolicited notification, not hangup and expect the
client to read it as a response to any new command it may issue.

I think it would be engineering mistake to send unsoliticated notifications
with the design presumption that the client will read this information.   It
only makes sense to send a 421 timeout when it is a response to a command
and YOU don't hang up.  Sure, you can send it, but it serves no purpose if
you are going to hangup anyway.  It is only useful for logging on your
server end.

May I ask what is the intent here with the unsolicited server notification?
I am not referring to a response to a request, but rather a notification
send to the client because it is idle in command mode, what is the client
suppose to do with it?

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 05 Jan 2004 17:44:55 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: what to say on timeout?
Sender: [email-masked]

At 16:47 05/01/2004, Richard O. Hammer wrote:

>Since the RFC provides for 421 on one kind of asynchronous shutdown, it 
>would seem consistent (and polite and professional and all that) to send 
>some error code in event of this other kind of asynchronous 
>shutdown.  There might be good arguments against sending such a message, 
>but I do not believe I have heard such arguments yet.

I don't think it really matters...
If you send such a 'response' asynchronously, then either the client has 
died somehow and won't see the response, so it won't matter, or it won't be 
listening for data and won't see the response so it won't matter, or it 
will see the response which will be useful.

Actually, my reading of RFC 2821 shows an inconsistency....

   --------
4.5.3.2 "An SMTP client MUST provide a timeout mechanism"

3.9 An SMTP server MUST NOT intentionally close the connection except:
-  After receiving a QUIT command and responding with a 221 reply.
-  After detecting the need to shut down the SMTP service and...
   --------

With a timeout, it's an intentional close of the connection, without 
receiving a QUIT command, and without detecting a need to shut down the 
SMTP *service* (at least, not my understanding of what that means)

If it *IS* shutting down the SMTP service, (in which case 3.9's 'MUST NOT' 
is meaningless), then the rest of 3.9 says that you SHOULD send a 421 
response asynchronously, whether the client is expecting it or not.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Mon, 05 Jan 2004 14:32:35 -0600
From: "Eric A. Hall" <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>
CC: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/5/2004 1:40 PM, Richard O. Hammer wrote:

> Hector Santos wrote:
> 
>>Send a 421 per RFC 2821 section 3.9.
> 
> One problem with 421 is that it normally means something else: I 
> believe that 421 normally means that the SMTP server is shutting down 
> for some reason.

That's the interpretation that is explicitly endorsed by the text in
RFC2821, but in the general sense, 421 just means "service unavailable",
which can include the (stated) case of critical failure, but can also mean
"oops, i'm out of disk space", or the ultimately generic "i'm bored with
this coversation". The client impact is pretty much the same in all of
these cases -- requeue the message(s) for delivery -- and it doesn't
really matter why. I suppose somebody could make the argument that a
server that is going down is probably going to be down for a while and
that another server could (or even should) be tried, but this isn't stated
anywhere in the algorithms and its not very pure logic, since a service
could just be termporarily recycled, or temp space could be freed, or
whatever, and those kinds of counter-scenarios nuke that argument.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Mon, 05 Jan 2004 14:40:57 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

Hector Santos wrote:
> Send a 421 per RFC 2821 section 3.9.

One problem with 421 is that it normally means something else: I 
believe that 421 normally means that the SMTP server is shutting down 
for some reason.  But a server which times-out a connection is not 
shutting down -- it is still operating and open for business.  So for 
such a server to send a 421 is a little bit of a lie, and may be 
misinterpreted by a client.

> ...   This is about idle timeouts (long inactivity) right?

Yes.  This is about when a server is sitting and waiting for the next 
command from the client, for some number of minutes (This could be a 
denial of service attack).  And it is also about too slow transmission 
of data after the DATA command.  If for instance the client sends only 
one packet every two minutes, and each packet contains only one 
additional byte of data, that could be cause for the server to timeout 
after some amount of time (A denial of service attack could also take 
this form).

> What I am saying if you are going to hang up due to long activity, the only
> thing that is useful is the response text.   The code itself is meaningless
> since the SMTP client would not know what to do with it.

Thank you for what you imply, but do not say directly, here.  You 
imply that the RFC does not tell what the server should say in this 
case.  You also imply that there is no standard or common practice for 
what the server should say in this case, because otherwise the client 
would know what to do.

Can you say -- forthrightly with confidence -- what you imply? that 
there is no formalized or standard message for the SMTP server to send 
just before it exercises its option to timeout because of long 
inactivity or too slow data transmission?

Thank you,
Rich

From: "Hector Santos" <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>, <[email-masked]>
Subject: Re: what to say on timeout?
Date: Mon, 5 Jan 2004 13:43:49 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: <[email-masked]>
Sent: Monday, January 05, 2004 11:47 AM
Subject: Re: what to say on timeout?

>
> Hector Santos wrote:
> > ... The client drives the server. Not
> > the other way around.
>
> Yes, I agree.  I have not suggested that the whole SMTP be changed
> from synchronous to asynchronous.  I ask only if the SMTP server
> should send a message with a particular error code in the circumstance
> where a server decides to terminate a connection because of inactivity.

Send a 421 per RFC 2821 section 3.9.

What I am saying is if the server is going to hang up anyway, it doesn't
matter.  This is about idle timeouts (long inactivity) right?  In which case
the SMTP client is not in a pending wait state or waiting for any command it
can work with.   If this was an interactive client, then I can only see a
usefulness if the server was going to send a "warning" the client to wake up
and do something, in which case, we don't have an appropiate client based
"noop" response code, an unsolicated information-only response code.

> Since the RFC provides for 421 on one kind of asynchronous shutdown,
> it would seem consistent (and polite and professional and all that) to
> send some error code in event of this other kind of asynchronous
> shutdown.  There might be good arguments against sending such a
> message, but I do not believe I have heard such arguments yet.

I hope I am not misconstrued suggesting no response should be provided.
What I am saying if you are going to hang up due to long activity, the only
thing that is useful is the response text.   The code itself is meaningless
since the SMTP client would not know what to do with it.  The response text
only usable in an interactive session and there again, maybe a 1 minute
warning can be sent first informing the full-duplex client to wake up.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 05 Jan 2004 13:37:08 -0500
From: "Richard O. Hammer" <[email-masked]>
To: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution
Sender: [email-masked]

Hector Santos wrote:
> In regards to RFC2821, section 5, saying:
>  ... 
> There seems to be a conflict between this and the following itemized rule in
> RFC1123:
>       5.3.4  Reliable Mail Transmission
>          ...
>          (1)  Multiple MX Records -- these contain a preference
>               indication that should be used in sorting.  If there are
>               multiple destinations with the same preference and there
>               is no clear reason to favor one (e.g., by address
>               preference), then the sender-SMTP SHOULD pick one at
>               random to spread the load across multiple mail exchanges
>               for a specific organization; note that this is a
>               refinement of the procedure in [DNS:3].
> 
> RFC2821 seems to suggest to include all equal preference MX records, while
> RFC1123 seems to suggest, unless I am reading it wrong, to choose 1 (at
> random) of the equal preference MX records.

I think RFC 1123 is ambiguous here.  You are right that it says to 
choose one.  But note that it does not say what to do next -- if the 
first one picked fails -- and that is the question.  Your 
interpretation can follow from what it says.

One possible interpretation (which I believe is yours) is:
1.  to skip over all the remaining destinations with the same 
preference (that is to pick one -- and only one -- destination with 
this preference).

But other interpretations can also follow, such as:
2. to pick another one at random from the remaining destinations with 
the same preference as the first one that failed (that is to pick 
first one at random, and then another one at random if the first fails).

Rich

From: "Hector Santos" <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: what to say on timeout?
Date: Mon, 5 Jan 2004 13:26:43 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Eric A. Hall" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: <[email-masked]>
Sent: Monday, January 05, 2004 12:46 PM
Subject: Re: what to say on timeout?

> A better example of this can be seen in negotiated TELNET, where the
> end-points have to progress through an ordered state path, but can do
> whatever they want whenever they want as long as the transaction is legal
> within the context of the current state. An even better example of this
> can be found in TCP itself.

> Regardless, end-points must be prepared to receive unexpected data and
> react accordingly. At the very least, an end-point should be prepared to
> find unexpected data whenever the input is read.

Right, which I think is where all this is coming from. :-)  Interactive
sessions or applications is where is "idle time out, shutdown pending"
notice will be useful.   Unless the client was going to "do something" and
also given a chance to wake up (You have 1 minute to do something),  I see
as an non-issue.  I  believe the original question was about idle timeouts
(long inactivity), in which case the client is not in a pending wait state.

Anyway, thanks for your input

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Mon, 05 Jan 2004 11:46:51 -0600
From: "Eric A. Hall" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

On 1/4/2004 11:33 PM, Hector Santos wrote:

> Unless I misunderstanding,  the above is based on the idea that the
> CLIENT has already sent a command and is currently waiting for a server
> response.
> 
> I don't know if any internet client protocol, maybe with the exception
> of FTP, where it can get a unsolicated server response.  A standard 
> client/server protocol state machine is simply not designed for this. 
> Clients are not going to wait around for something it is not expecting.
> It issues a command, waits for a response.  The client drives the
> server. Not the other way around.

By default, SMTP sessions follow a synchonous state path, which further
consists of synchronous, half-duplex transaction pairs (the <CRLF> and/or
'.' at the end of input is essentially a Go-Ahead signal). Pipelining
partly allows the transaction pairs to be full-duplex, but does not
otherwise affect the synchronicity of the transaction pairs or the overall
state machine.

As was pointed out already, IMAP allows full-duplex, asynchronous
transaction pairs (although its less common than might be expected in
practice, due to certain implications with the transaction structure)
while the overall state path for IMAP sessions are still synchronous.

A better example of this can be seen in negotiated TELNET, where the
end-points have to progress through an ordered state path, but can do
whatever they want whenever they want as long as the transaction is legal
within the context of the current state. An even better example of this
can be found in TCP itself.

Regardless, end-points must be prepared to receive unexpected data and
react accordingly. At the very least, an end-point should be prepared to
find unexpected data whenever the input is read.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Mon, 05 Jan 2004 11:47:40 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

Hector Santos wrote:
> ... The client drives the server. Not
> the other way around.

Yes, I agree.  I have not suggested that the whole SMTP be changed 
from synchronous to asynchronous.  I ask only if the SMTP server 
should send a message with a particular error code in the circumstance 
where a server decides to terminate a connection because of inactivity.

Now I agree that it is asynchronous if the server sends a timeout 
error message without waiting to send it in response to a client 
command.  But it is also asynchronous to do what you Hector suggested 
earlier in the thread, to "Simply log the timeout and drop the 
connection" without waiting for a QUIT from the client, and it is also 
asynchronous to do what the RFC says about sending 421 when server 
shutdown is forced for some reason.  So asynchronous shutdown actions 
are already practiced as you suggest, and already specified in the RFC.

RFC 2821 Section 4.5.3.2 allows for asynchronous SMTP server timeout. 
  But unless I am mistaken the RFC does not tell whether a server, 
when exercising this option, should send an error code message or drop 
the connection silently.  This is what I ask.

Since the RFC provides for 421 on one kind of asynchronous shutdown, 
it would seem consistent (and polite and professional and all that) to 
send some error code in event of this other kind of asynchronous 
shutdown.  There might be good arguments against sending such a 
message, but I do not believe I have heard such arguments yet.

Thank you,
Rich

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
Date: Mon, 5 Jan 2004 09:21:17 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Monday, January 05, 2004 4:01 AM
Subject: Re: RFC 2821 Address Resolution

> If you were *really* smart,  you recorded how long the first lookup took,
> and use that info to decide if you fire the second DNS when you start 7,
> when you start 8, or when you're N seconds from deciding that 8 is dead -
> if the first query took 2 seconds, you can wait till 3-4 seconds
> before you think you need to start 9, if the first query took 30
> seconds, you need to fire the DNS lookups 35-40 seconds ahead.

I wonder what would be the overall gain with this.  With the new resolver
expanding the list, it will need to be augmented with a system policy option
to define the number of maximum servers per session.  I probably will give
sysops a default of 1/2 the amount.  In my off the cuff view, if a system
has trouble sending mail by 1/2+1, well, that means to mean that the IP list
is faulty to being with, i.e., operational or scalability issue at the
remote end.   What do you think about this?  I agree with Richard, 80% of
the time, it is successful in the first few servers.

But you now made me wonder about it, the R&D hat in me will make me explore
it. <g>

> I have to give credit for much of that idea to Thom O'Connor and
> Motonori Nakamura, both of whom have spent a lot more time
> analyzing the DNS issues of SMTP for scaling high-volume delivery
> than I have.

bulk spammers are probably experts at it too <g>

> .....  In other words, if your mail is
> going to AOL and Yahoo, waiting for both sets of DNS queries to
> finish before you connect to anybody can kill your performance...

Right. No, our system is a 100% multi-threaded Windows system. The sysop can
define the maximum number of worker threads - highly optimized high end
server system. :-)

Thanks for your input Valdis.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Brad Clements" <[email-masked]>
Organization: MurkWorks, Incorporated.
To: IETF-SMTP <[email-masked]>
Date: Mon, 05 Jan 2004 09:33:04 -0500
Subject: Re: RFC 2821 Address Resolution 
Reply-to: [email-masked]
Content-description: Mail message body
Sender: [email-masked]

On 5 Jan 2004 at 1:35, [email-masked] wrote:

> OK.. I misread the table and my memory both.   When we first got on Bitnet
> in 1984, Milnet was ahead 1024-440, Bitnet took the lead during late 85 to
> early 86, and Milnet was again leading by 5K to 3K hosts when we multihomed
> that Microvax in late 86. We were still a bitnet-only site during the
> period where Bitnet was larger...
> 
> I suddenly feel old. :)
> 

Get in line. That very same Microvax is sitting in my garage. Would you like me to send 
it to you, freight collect?

-- 
Brad Clements,                [email-masked]   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
http://www.wecanstopspam.org/                   AOL-IM: BKClements

From: Arnt Gulbrandsen <[email-masked]>
To: Hector Santos <[email-masked]>
Subject: Re: what to say on timeout?
Cc: [email-masked]
Date: Mon, 5 Jan 2004 11:39:11 +0100
Sender: [email-masked]

Hector Santos writes:
> Unless I misunderstanding, the above is based on the idea that the 
> CLIENT has already sent a command and is currently waiting for a 
> server response.

The client will see the 421 at once, or when it sends its next command, 
depending on how it's written. That's why 421 is legal for all 
commands.

> I don't know if any internet client protocol, maybe with the exception 
> of FTP, where it can get a unsolicated server response.

IMAP, for example. And there are many internet protocols that don't work 
well with an RPC client like this:

>          SendCommand("HELO")
>                 WaitForResponse()    // Synchronous concept
>          SendCommand("MAIL FROM:")
>                 WaitForResponse()    // Synchronous concept

Read the SMTP PIPELINING RFC, look at the NNTP client code in Leafnode, 
look at the client part of any fast NFS implementation, etc.

Fast clients may look like this:

     when necessary and possible:
         send a command
     when data arrives from the server:
         figure out which command it belongs to
         update the client's state machine

--Arn

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2821 Address Resolution
Cc: Hector Santos <[email-masked]>, [email-masked]
Date: Mon, 5 Jan 2004 11:01:17 +0100
Sender: [email-masked]

[email-masked] writes:
> Oh, and another not-yet-mentioned piece of wisdom from Motonori:  When 
> dealing with delivery of a message to multiple sites, having a truly 
> asynchronous multi-threaded resolver is an incredible win (if the 
> mail is going to 40 places, launching all 40 DNS queries at once 
> means that you can start delivering mail as soon as you get the FIRST 
> DNS reply packet back - even if you're doing MX piggybacking(*), 
> you're still in the clear as long as any additional replies 
> indicating piggybacking arrive in time for you to merge in their RCPT 
> TO: (basically, as long as you haven't gotten to the DATA command).

You can improve that another bit if you a) delay the SMTP connection for 
a few tenths of a second from the time of the first DNS reply and b) 
base piggybacking on the IP address of each SMTP server rather than MX 
name. In my experience, at least.

--Arnt

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
From: [email-masked]
Date: Mon, 05 Jan 2004 04:01:15 -0500
Sender: [email-masked]

--==_Exmh_673435022P
Content-Type: text/plain; charset=us-ascii

On Mon, 05 Jan 2004 03:26:08 EST, Hector Santos said:

> count : 11
> MX     0  | 15 | 205.188.156.185  | mailin-01.mx.aol.com |
> MX     1  | 15 | 205.188.158.121  | mailin-01.mx.aol.com |
> MX     2  | 15 | 205.188.159.57   | mailin-01.mx.aol.com |
> MX     3  | 15 | 64.12.137.89     | mailin-01.mx.aol.com |
> MX     4  | 15 | 64.12.137.184    | mailin-01.mx.aol.com |
> MX     5  | 15 | 64.12.138.57     | mailin-01.mx.aol.com |
> MX     6  | 15 | 64.12.138.152    | mailin-01.mx.aol.com |
> MX     7  | 15 | 64.12.138.89     | mailin-02.mx.aol.com |
> MX     8  | 15 | 64.12.138.120    | mailin-02.mx.aol.com |
> MX    *9  | 15 | 0.0.0.0          | mailin-04.mx.aol.com |
> MX    *10 | 15 | 0.0.0.0          | mailin-03.mx.aol.com |
> 
> Here there are 4 MX with equal preference.  mailin-04 and mailin-03 has no A
> records, so they end up at towards the bottom of the final list (indicated
> with the asterisk).    So what will happen in the sender is that if he sees
> an IP of zero, it will then perform the A record lookup at the point.
> 
> I believe this is the "clever" logic you suggested.  Right?

That's a good first cut.  The "even more clever" version is to fire off your
second round of DNS lookups when you get to entry 7 or 8 so that you get
the answer back in time to start trying the yet-to-be-filled-in entry at 9
without delaying 9, and without wasting resources if one of the first 6 or 7
entries in the table works.  If you were *really* smart, you recorded how long
the first lookup took, and use that info to decide if you fire the second DNS
when you start 7, when you start 8, or when you're N seconds from deciding
that 8 is dead - if the first query took 2 seconds, you can wait till 3-4 seconds
before you think you need to start 9, if the first query took 30 seconds, you need
to fire the DNS lookups 35-40 seconds ahead.

I have to give credit for much of that idea to Thom O'Connor and Motonori
Nakamura, both of whom have spent a lot more time analyzing the DNS issues of
SMTP for scaling high-volume delivery than I have.

Oh, and another not-yet-mentioned piece of wisdom from Motonori:  When dealing
with delivery of a message to multiple sites, having a truly asynchronous
multi-threaded resolver is an incredible win (if the mail is going to 40
places, launching all 40 DNS queries at once means that you can start
delivering mail as soon as you get the FIRST DNS reply packet back - even if
you're doing MX piggybacking(*), you're still in the clear as long as any
additional replies indicating piggybacking arrive in time for you to merge in
their RCPT TO: (basically, as long as you haven't gotten to the DATA command).
In other words, if your mail is going to AOL and Yahoo, waiting for both sets of
DNS queries to finish before you connect to anybody can kill your performance...

--==_Exmh_673435022P
Content-Type: application/pgp-signature

--==_Exmh_673435022P--

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
Date: Mon, 5 Jan 2004 03:26:08 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Sunday, January 04, 2004 4:31 PM
Subject: Re: RFC 2821 Address Resolution

> > 1) It's unclear if you should try the first mx1, the first mx2,
> > the second mx1, the second mx2, or first try all the mx1 and then
> > all the mx2.

>> Group them and then randomized?

> Certainly *not* - you're required to preserve per-host ordering of the A
> records (as you don't KNOW if they're round-robined, or if they are
> preference-ordered - the first could have an OC12 and the 3rd or 4th A
> record correspond to a DS3).

Sorry, I mean randomize the MX records.

What I ended up doing for my new resolver to minimize the lookup.  So if MX1
and MX2 are equal preference but one of them do no have any A records in the
MX query, then add the MX to the end of the list (before any higher
preference entry) with no A records.   This will allow for the MX entries
with A records to be tried first.

Here is an example output using AOL.COM with a UDP-based DNS MX lookup
resulting with incomplete information and how I finalize the list:

-- [aol.com] --
DNS> records  : 22 an: 4  ns: 4  ar: 14
DNS> RR MX   ttl=636    pre=15  | aol.com | mailin-03.mx.aol.com
DNS> RR MX   ttl=636    pre=15  | aol.com | mailin-04.mx.aol.com
DNS> RR MX   ttl=636    pre=15  | aol.com | mailin-01.mx.aol.com
DNS> RR MX   ttl=636    pre=15  | aol.com | mailin-02.mx.aol.com
DNS> RR A    ttl=135    | 205.188.156.185  | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 205.188.158.121  | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 205.188.159.57   | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 64.12.137.89     | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 64.12.137.184    | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 64.12.138.57     | mailin-01.mx.aol.com
DNS> RR A    ttl=135    | 64.12.138.152    | mailin-01.mx.aol.com
DNS> RR A    ttl=103    | 64.12.137.184    | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 64.12.138.89     | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 64.12.138.120    | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 205.188.156.185  | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 205.188.158.121  | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 205.188.159.57   | mailin-02.mx.aol.com
DNS> RR A    ttl=103    | 64.12.137.89     | mailin-02.mx.aol.com

count : 11
MX     0  | 15 | 205.188.156.185  | mailin-01.mx.aol.com |
MX     1  | 15 | 205.188.158.121  | mailin-01.mx.aol.com |
MX     2  | 15 | 205.188.159.57   | mailin-01.mx.aol.com |
MX     3  | 15 | 64.12.137.89     | mailin-01.mx.aol.com |
MX     4  | 15 | 64.12.137.184    | mailin-01.mx.aol.com |
MX     5  | 15 | 64.12.138.57     | mailin-01.mx.aol.com |
MX     6  | 15 | 64.12.138.152    | mailin-01.mx.aol.com |
MX     7  | 15 | 64.12.138.89     | mailin-02.mx.aol.com |
MX     8  | 15 | 64.12.138.120    | mailin-02.mx.aol.com |
MX    *9  | 15 | 0.0.0.0          | mailin-04.mx.aol.com |
MX    *10 | 15 | 0.0.0.0          | mailin-03.mx.aol.com |

Here there are 4 MX with equal preference.  mailin-04 and mailin-03 has no A
records, so they end up at towards the bottom of the final list (indicated
with the asterisk).    So what will happen in the sender is that if he sees
an IP of zero, it will then perform the A record lookup at the point.

I believe this is the "clever" logic you suggested.  Right?

Seems to work with over 500 domains I tested it with.   Here is a another
snapshot:

-- [aol.com] --
DNS> records  : 22 an: 4  ns: 4  ar: 14
DNS> RR MX   ttl=439    pre=15  | aol.com | mailin-01.mx.aol.com
DNS> RR MX   ttl=439    pre=15  | aol.com | mailin-02.mx.aol.com
DNS> RR MX   ttl=439    pre=15  | aol.com | mailin-03.mx.aol.com
DNS> RR MX   ttl=439    pre=15  | aol.com | mailin-04.mx.aol.com
DNS> RR A    ttl=222    | 64.12.138.89     | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 64.12.138.120    | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 205.188.156.185  | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 205.188.158.121  | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 205.188.159.57   | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 64.12.137.89     | mailin-02.mx.aol.com
DNS> RR A    ttl=222    | 64.12.137.184    | mailin-02.mx.aol.com
DNS> RR A    ttl=76     | 64.12.137.121    | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 64.12.137.152    | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 64.12.138.57     | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 64.12.138.120    | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 205.188.156.249  | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 205.188.157.25   | mailin-03.mx.aol.com
DNS> RR A    ttl=76     | 205.188.159.217  | mailin-03.mx.aol.com

count : 15
MX     0  | 15 | 64.12.138.89     | mailin-02.mx.aol.com |
MX     1  | 15 | 64.12.138.120    | mailin-02.mx.aol.com |
MX     2  | 15 | 205.188.156.185  | mailin-02.mx.aol.com |
MX     3  | 15 | 205.188.158.121  | mailin-02.mx.aol.com |
MX     4  | 15 | 205.188.159.57   | mailin-02.mx.aol.com |
MX     5  | 15 | 64.12.137.89     | mailin-02.mx.aol.com |
MX     6  | 15 | 64.12.137.184    | mailin-02.mx.aol.com |
MX     7  | 15 | 64.12.137.121    | mailin-03.mx.aol.com |
MX     8  | 15 | 64.12.137.152    | mailin-03.mx.aol.com |
MX     9  | 15 | 64.12.138.57     | mailin-03.mx.aol.com |
MX     10 | 15 | 205.188.156.249  | mailin-03.mx.aol.com |
MX     11 | 15 | 205.188.157.25   | mailin-03.mx.aol.com |
MX     12 | 15 | 205.188.159.217  | mailin-03.mx.aol.com |
MX    *13 | 15 | 0.0.0.0          | mailin-04.mx.aol.com |
MX    *14 | 15 | 0.0.0.0          | mailin-01.mx.aol.com |

This time mailin-01 and mailin-04 end up in the bottom.    Notice list is
larger.  In the first list, duplicate IPs were removed.

Here is YAHOO.COM which offers 3 MX host with 2 equal preferences:

-- [yahoo.com] --
DNS> records  : 21 an: 3  ns: 5  ar: 13
DNS> RR MX   ttl=1583   pre=1   | yahoo.com | mx2.mail.yahoo.com
DNS> RR MX   ttl=1583   pre=5   | yahoo.com | mx4.mail.yahoo.com
DNS> RR MX   ttl=1583   pre=1   | yahoo.com | mx1.mail.yahoo.com
DNS> RR A    ttl=61     | 64.157.4.79      | mx1.mail.yahoo.com
DNS> RR A    ttl=61     | 64.156.215.5     | mx1.mail.yahoo.com
DNS> RR A    ttl=61     | 64.156.215.6     | mx1.mail.yahoo.com
DNS> RR A    ttl=61     | 64.156.215.7     | mx1.mail.yahoo.com
DNS> RR A    ttl=61     | 64.157.4.78      | mx1.mail.yahoo.com
DNS> RR A    ttl=1292   | 64.157.4.78      | mx2.mail.yahoo.com
DNS> RR A    ttl=1292   | 64.156.215.5     | mx2.mail.yahoo.com
DNS> RR A    ttl=1292   | 64.156.215.6     | mx2.mail.yahoo.com
DNS> RR A    ttl=120    | 216.155.197.63   | mx4.mail.yahoo.com
DNS> RR A    ttl=120    | 66.218.86.156    | mx4.mail.yahoo.com
DNS> RR A    ttl=120    | 66.218.86.253    | mx4.mail.yahoo.com
DNS> RR A    ttl=120    | 66.218.86.254    | mx4.mail.yahoo.com
DNS> RR A    ttl=120    | 67.28.114.32     | mx4.mail.yahoo.com

count : 10
MX     0  | 1  | 64.157.4.78      | mx2.mail.yahoo.com |
MX     1  | 1  | 64.156.215.5     | mx2.mail.yahoo.com |
MX     2  | 1  | 64.156.215.6     | mx2.mail.yahoo.com |
MX     3  | 1  | 64.157.4.79      | mx1.mail.yahoo.com |
MX     4  | 1  | 64.156.215.7     | mx1.mail.yahoo.com |
MX     5  | 5  | 216.155.197.63   | mx4.mail.yahoo.com |
MX     6  | 5  | 66.218.86.156    | mx4.mail.yahoo.com |
MX     7  | 5  | 66.218.86.253    | mx4.mail.yahoo.com |
MX     8  | 5  | 66.218.86.254    | mx4.mail.yahoo.com |
MX     9  | 5  | 67.28.114.32     | mx4.mail.yahoo.com |

and another:

-- [yahoo.com] --
DNS> records  : 21 an: 3  ns: 5  ar: 13
DNS> RR MX   ttl=1378   pre=1   | yahoo.com | mx1.mail.yahoo.com
DNS> RR MX   ttl=1378   pre=1   | yahoo.com | mx2.mail.yahoo.com
DNS> RR MX   ttl=1378   pre=5   | yahoo.com | mx4.mail.yahoo.com
DNS> RR A    ttl=1722   | 64.157.4.79      | mx1.mail.yahoo.com
DNS> RR A    ttl=1722   | 64.156.215.5     | mx1.mail.yahoo.com
DNS> RR A    ttl=1722   | 64.156.215.6     | mx1.mail.yahoo.com
DNS> RR A    ttl=1722   | 64.156.215.7     | mx1.mail.yahoo.com
DNS> RR A    ttl=1722   | 64.157.4.78      | mx1.mail.yahoo.com
DNS> RR A    ttl=1087   | 64.156.215.5     | mx2.mail.yahoo.com
DNS> RR A    ttl=1087   | 64.156.215.6     | mx2.mail.yahoo.com
DNS> RR A    ttl=1087   | 64.157.4.78      | mx2.mail.yahoo.com
DNS> RR A    ttl=1721   | 66.218.86.253    | mx4.mail.yahoo.com
DNS> RR A    ttl=1721   | 66.218.86.254    | mx4.mail.yahoo.com
DNS> RR A    ttl=1721   | 67.28.114.32     | mx4.mail.yahoo.com
DNS> RR A    ttl=1721   | 216.155.197.63   | mx4.mail.yahoo.com
DNS> RR A    ttl=1721   | 66.218.86.156    | mx4.mail.yahoo.com

count : 10
MX     0  | 1  | 64.157.4.79      | mx1.mail.yahoo.com |
MX     1  | 1  | 64.156.215.5     | mx1.mail.yahoo.com |
MX     2  | 1  | 64.156.215.6     | mx1.mail.yahoo.com |
MX     3  | 1  | 64.156.215.7     | mx1.mail.yahoo.com |
MX     4  | 1  | 64.157.4.78      | mx1.mail.yahoo.com |
MX     5  | 5  | 66.218.86.253    | mx4.mail.yahoo.com |
MX     6  | 5  | 66.218.86.254    | mx4.mail.yahoo.com |
MX     7  | 5  | 67.28.114.32     | mx4.mail.yahoo.com |
MX     8  | 5  | 216.155.197.63   | mx4.mail.yahoo.com |
MX     9  | 5  | 66.218.86.156    | mx4.mail.yahoo.com |

Seems to be doing the job now.

Thanks for your input.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

To: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
From: [email-masked]
Date: Mon, 05 Jan 2004 01:35:52 -0500
Sender: [email-masked]

--==_Exmh_36082400P
Content-Type: text/plain; charset=us-ascii

On Mon, 05 Jan 2004 00:43:05 EST, [email-masked] said:

> But what do I know?  I once helped multihome a machine on Bitnet and Arpanet/
> Milnet, and Bitnet was the larger network at the time.. ;)

OK.. I misread the table and my memory both.   When we first got on Bitnet in
1984, Milnet was ahead 1024-440, Bitnet took the lead during late 85 to early
86, and Milnet was again leading by 5K to 3K hosts when we multihomed that
Microvax in late 86. We were still a bitnet-only site during the period where
Bitnet was larger...

I suddenly feel old. :)

--==_Exmh_36082400P
Content-Type: application/pgp-signature

--==_Exmh_36082400P--

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
From: [email-masked]
Date: Mon, 05 Jan 2004 00:43:05 -0500
Sender: [email-masked]

--==_Exmh_-185790118P
Content-Type: text/plain; charset=us-ascii

On Mon, 05 Jan 2004 00:10:21 EST, Hector Santos said:

> In regards to RFC2821, section 5, saying:

> There seems to be a conflict between this and the following itemized rule in
> RFC1123:

The world has moved on since 1123 was written.  I don't think in October 1989,
*ANYBODY* had the sort of mail infrastructure where multiple MXs and multiple As
were in use to load-balance.  It was strictly "One MX with one A record for our
on-site server, and another MX pointing to a single A record offsite".  I remember
at that time, Marc Crispin was an oddity because he had 4 MX's defined, all on
separate continents...

http://www.isc.org/ds/host-count-history.html  says that October 1989, there
were 159,000 hosts. Total.  That's only 5 times what *my site* has today.

But what do I know?  I once helped multihome a machine on Bitnet and Arpanet/
Milnet, and Bitnet was the larger network at the time.. ;)

--==_Exmh_-185790118P
Content-Type: application/pgp-signature

--==_Exmh_-185790118P--

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: what to say on timeout?
Date: Mon, 5 Jan 2004 00:33:01 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: <[email-masked]>
Sent: Sunday, January 04, 2004 10:47 PM
Subject: Re: what to say on timeout?

> But there is the 421 message which SMTP servers can send.  RFC 2821,
> Section 3.9 says:
>
>     An SMTP server MUST NOT intentionally close the connection except:
>  ...
>     -  After detecting the need to shut down the SMTP service and
>        returning a 421 response code.  This response code can be issued
>        after the server receives any command or, if necessary,
>        asynchronously from command receipt (on the assumption that the
>        client will receive it after the next command is issued).
>  ...
>
> It seems to me that timeout because of inactivity has some similarity
> with the circumstance calling for a 421 response code.  In both cases
> the server is going to terminate the connection.  If it is expected to
> send a response code asynchronously from command receipt in the case
> of forced server shutdown, why not in the case of timeout?

Because the client is not expecting it.

Unless I misunderstanding,  the above is based on the idea that the CLIENT
has already sent a command and is currently waiting for a server response.

I don't know if any internet client protocol, maybe with the exception of
FTP, where it can get a unsolicated server response.  A standard
client/server protocol state machine is simply not designed for this.
Clients are not going to wait around for something it is not expecting.  It
issues a command, waits for a response.  The client drives the server. Not
the other way around.

It is outlined like this:

         SendCommand("HELO")
                WaitForResponse()    // Synchronous concept
         SendCommand("MAIL FROM:")
                WaitForResponse()    // Synchronous concept
         SendCommand("RCPT TO:")
                WaitForResponse()    // Synchronous concept
         SendCommand("DATA")
                WaitForResponse()    // Synchronous concept
         SendCommand("QUIT")
                WaitForResponse()    // Synchronous concept

etc.  It is not an asynchronous concept such as:

         SendCommand("HELO")
                WaitForResponse()    // Synchronous concept

         CheckForAnyMoreServerResponse()    // /Asynchronous Concept
         SendCommand("MAIL FROM:")
                WaitForResponse()    // Synchronous concept

         CheckForAnyMoreServerResponse()    // /Asynchronous Concept
         SendCommand("RCPT TO:")
                WaitForResponse()    // Synchronous concept

         CheckForAnyMoreServerResponse()    // /Asynchronous Concept
         SendCommand("DATA")
                WaitForResponse()    // Synchronous concept

         CheckForAnyMoreServerResponse()    // /Asynchronous Concept
         SendCommand("QUIT")
                WaitForResponse()    // Synchronous concept

Now, if that is what is being suggested here, then that would completely new
to me and an odd, non-standard client/server design methodology.

The only thing close the above is maybe this:

         SendCommand("HELO")
                WaitForResponse()    // Synchronous concept

         CheckForSocketAlive()    // /Asynchronous Concept
         SendCommand("MAIL FROM:")
                WaitForResponse()    // Synchronous concept

         CheckForSocketAlive()    // /Asynchronous Concept
         SendCommand("RCPT TO:")
                WaitForResponse()    // Synchronous concept

         CheckForSocketAlive()    // /Asynchronous Concept
         SendCommand("DATA")
                WaitForResponse()    // Synchronous concept

         CheckForSocketAlive()    // /Asynchronous Concept
         SendCommand("QUIT")
                WaitForResponse()    // Synchronous concept

Which can be already done at the SendCommand() with an implicit socket close
error on the send() command.

Right?

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
Date: Mon, 5 Jan 2004 00:10:21 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Sunday, January 04, 2004 2:09 PM
Subject: Re: RFC 2821 Address Resolution

Valdis (and Richard),

In regards to RFC2821, section 5, saying:

   Multiple MX records contain a preference indication that MUST be used
   in sorting (see below).  Lower numbers are more preferred than higher
   ones.  If there are multiple destinations with the same preference
   and there is no clear reason to favor one (e.g., by recognition of an
   easily-reached address), then the sender-SMTP MUST randomize them to
   spread the load across multiple mail exchangers for a specific
   organization.

There seems to be a conflict between this and the following itemized rule in
RFC1123:

RFC1123  "Requirements for Internet Hosts -- Application and Support"

      5.3.4  Reliable Mail Transmission

         ...

         The following information is to be used to rank the host
         addresses:

         (1)  Multiple MX Records -- these contain a preference
              indication that should be used in sorting.  If there are
              multiple destinations with the same preference and there
              is no clear reason to favor one (e.g., by address
              preference), then the sender-SMTP SHOULD pick one at
              random to spread the load across multiple mail exchanges
              for a specific organization; note that this is a
              refinement of the procedure in [DNS:3].

RFC2821 seems to suggest to include all equal preference MX records, while
RFC1123 seems to suggest, unless I am reading it wrong, to choose 1 (at
random) of the equal preference MX records.     [DNS.3] references RFC974
which is more consistent with RFC2821.

What say you?

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Sun, 04 Jan 2004 22:47:11 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

Hector Santos wrote:
> SMTP is a client/server state machine protocol .   Like in most state
> machines,  the SMTP server should not send out a response or information
> unless it received a request, command or signal (as in the SMTP connection
> signal) from the client. SMTP clients are not designed to received
> unsolicited responses.

But there is the 421 message which SMTP servers can send.  RFC 2821, 
Section 3.9 says:

    An SMTP server MUST NOT intentionally close the connection except:
	...
    -  After detecting the need to shut down the SMTP service and
       returning a 421 response code.  This response code can be issued
       after the server receives any command or, if necessary,
       asynchronously from command receipt (on the assumption that the
       client will receive it after the next command is issued).
	...

    An SMTP server which is forcibly shut down via external means SHOULD
    attempt to send a line containing a 421 response code to the SMTP
    client before exiting.  The SMTP client will normally read the 421
    response code after sending its next command.

It seems to me that timeout because of inactivity has some similarity 
with the circumstance calling for a 421 response code.  In both cases 
the server is going to terminate the connection.  If it is expected to 
send a response code asynchronously from command receipt in the case 
of forced server shutdown, why not in the case of timeout?

Rich

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution
Date: Sun, 4 Jan 2004 22:20:17 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Sent: Sunday, January 04, 2004 3:23 PM
Subject: Re: RFC 2821 Address Resolution

> Just to make sure we are using the same language, a "host" is a name
> like mailin-02.mx.aol.com which you get in an MX record.  ....

Thanks for the clarification of the technical semantics; domain, host and
home. I agree it is useful to be consistent here.  But please excuse me if I
use domain, MX and IP (A record) here and there. :-)

> Hector Santos wrote:
>  > The logic I used is:
>  >
>  > 1) Sort the MX records (low to high) preference,
>  > 2) For each MX, add the A records to the final list removing
> duplicate ips.
>
> That sounds right to me.  My code does not look for or remove
> duplicate IPs; I do not remember seeing this suggested in the RFC, but
> it makes sense.

Thanks. The technical question of timing came into play which I see you
address reading the rest of the message.

> ...
>
> Further, the list of A-records appended to a MX query does not
> pretend to be complete.  The lists which you show appear to me to be
> truncated, perhaps at a size limit determined by the packet size.

Yes, I finalized realized this by switching to a stream lookup where all the
information came in.

> An optimization: Since there are potentially many homes (maybe
> sometimes even 100) for a given domain, and since my logs indicate to
> me that the first attempt to connect to any home usually succeeds, 80%
> of the time or better, it may be wasteful to prepare the entire list
> of homes, sorted by host-priority, before attempting to connect to the
> first home in the list.  The first attempt will probably succeed and
> make the rest of the list useless.  Initially, my code does no more
> DNS lookups than necessary to find the first home that it should
> attempt for a domain; it postpones further DNS lookups and sorting
> until needed, if ever.

Thanks, this seems to be the final conclusion to this exercise. :-)

I went back to see our original 1996 code and I see that it did a simple MX
(host) lookup, sorted them and then it used gethostbyname() when the sorted
MX was attempted by the sender.   Incorrect by RFC standards because it was
only using the first IP returned by gethostbyname(), not the entire list,
but it was more dynamic.  If it had to go to the second MX, the fresh lookup
would give more current information.

So the only real change here is to have the sender lookup all the homes (ip
addresses) for the host making it only necessary to do when needed.

However, the initial MX lookup A record information should be used to
further minimize the initial A records lookup by the sender for the first
MX.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
Date: Sun, 4 Jan 2004 21:45:38 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Sunday, January 04, 2004 4:31 PM
Subject: Re: RFC 2821 Address Resolution

> What's happening with AOL is that they have a LARGE farm of
> servers that won't fit into a single DNS reply packet, so they
> give you the complete MX list, and hopefully enough A records
> that you'll find a working one without having to do either a
> lookup for the A records for the other Max's, or set up a TCP
> connection to get the full DNS reply (with the 3 packet overhead
> at the front, the FIN/ACK at the end, and all the other ugly
> overhead-sucking stuff).

Right, I finally realized this after I got some sleep. I verified this by
switching to a stream connection. :-)

> A more subtle hole I've seen MTA's fall into - do the MX and A
> record lookups, get a list of 20 or so A records with a 30-min
> TTL.  Start trying, 2 minute timeout on each.  When you get to
> the 16th and fail to notice it's stale and needs to be looked up
> again.

Why not delay the lookup until the moment the MX will be used by the sender?
This can address the problem where maybe the first set of MX address(es)
failed to connect or timed out, making the secondary set somewhat expired.
Hmmmmm, but then again, that could make the order of MX expired as well.

Anyway, the lesson learn for us is that we were incorrectly limiting the set
of A records to try for single MX, multi-home
systems.

The original 1996 design simply looked up the MX records, sorted them by
preference. The send mail thread performed the gethostbyname() at the moment
the MX was attempted.   Changes over the years included getting
the IP immediately by calling gethostbyname() during the MX lookup,
excluding dupes and also adding support for multiple DNS server lookup
usage.

Sounds to me that only change I need to do is to revert back to the original
sorted MX lookup, then change the send mail logic to retrieve all the IPs
for the MX.   This should minimize the lookups.  But then again, my new
resolver utilizing the A records in the MX lookup RDATA dataset seems to be
appropriate too WHEN combined with an option to set a minimum amount
desired.  So for systems like AOL.COM returning at least 10 records of a
possible 16 or more, would be sufficient for most operations.

Thanks for your input.  Learned a few things. :-)

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
From: [email-masked]
Date: Sun, 04 Jan 2004 16:31:38 -0500
Sender: [email-masked]

--==_Exmh_1106085364P
Content-Type: text/plain; charset=us-ascii

On Sun, 04 Jan 2004 15:32:09 EST, Hector Santos said:

> So an inquiring client should "randomize" it (again) to  minimize the
> possibility that two or more current client snapshots are not going to hit
> the same servers at the same time?

You need to randomize the list of equal-weight MX's, but leave the order
of A records for each MX alone.

> > In fact, a single client can't even tell round-robin is in use
> > unless you ask several times (I once had the luck of doing a 'dig'
> > of a address I *knew* was round-robined from the server listed
> > in the NS (so any caching at my end didn't matter), and I had to
> > ask *five times* before I got a different list. I just happened to
> > be the N'th querie 4 times in a row, so it had round-robined back
> > to where I was)...
> 
> This was what was happening with gethostbyname(). A direct A record lookup
> yield a round-robin result, but the Windows socket function resolved it from
> its cache first returning the same address over and over again until it
> timed out and fetched a new set of records.

No, you're seeing something different.  I in fact queried the NS *directly*,
but they had a list of 5 or so round-robin'ed addresses, I'd do the query, some
(5*N)-1 queries would arrive, my next query would show up, and it would be at
the same place again. Same thing happened a few more times, till I got in at
some query number other than (5*N)-1. ;) Kind of like a 'Wheel of Fortune'
wheel, and hitting "Lose Your Turn" several times in a row... :)

> > Yes, that's sub-optimal if you cached a list that has the downed
> > host first in a list of 4, but hopefully only 1/4 of the hosts
> > end up in that situation.

> For aol.com, I'm noticing it is usually 2 of the 4 that are down (no A
> record in the MX query) with round-robin results.

"No A record" is different than "down".  I mean the case where you get back
a list of 4 round-robin'ed A's, and the first one is down for maintenance, so you keep
trying that one and then re-trying onto the second A record.  And you keep on
doing that till the TTL on the entry expires and you refetch - even though you'd
have gotten out of that situation (with 75% probability) if you had ignored the TTL
and re-fetched...

What's happening with AOL is that they have a LARGE farm of servers that won't
fit into a single DNS reply packet, so they give you the complete MX list, and hopefully
enough A records that you'll find a working one without having to do either a
lookup for the A records for the other MX's, or set up a TCP connection to get the
full DNS reply (with the 3 packet overhead at the front, the FIN/ACK at the end, and
all the other ugly overhead-sucking stuff).

Watch AOL for an extended period (the better part of a day) - you'll notice that
they slowly rotate MX and A records into and out of the list, so they can rely on
sites keeping older but still valid addresses in their cache while they start advertising
new ones - this allows them to *effectively* have a larger set of servers than they
could actually advertise in a single-packet UDP DNS response...

> But what do you do with multi-MX where one or more is down and have lower
> preference numbers than the rest?

It's pretty clear you need to exhaust all the options for each lower-number MX
before moving on.  So you do all the mx=0, then mx=1, then mx=2 and so on.

> > 1) It's unclear if you should try the first mx1, the first mx2,
> > the second mx1, the second mx2, or first try all the mx1 and then
> > all the mx2.

> Group them and then randomized?

Certainly *not* - you're required to preserve per-host ordering of the A
records (as you don't KNOW if they're round-robined, or if they are
preference-ordered - the first could have an OC12 and the 3rd or 4th A
record correspond to a DS3).

The question is basically "which is better, the second-best address for the
first MX, or the best address for the second one" - and given that the
"first" and "second" are required to be randomized, the best address of
the second is a better bet, even though the language in the RFC *hints*
at running all the A's for one, then the other.

> > 2) It's unclear if you encounter a duplicate address you've tried
> > already on ONE mx if you should remove it from the list, or
> > re-try if it's the other MX.

> I'm removing it from the list currently.  If the server is not responding,
> why try it again during the current send mail session?

Some further thought indicates it probably requires checking if your retry
timeout for a given host has expired. If you retry a given host every 30
minutes, and allow a 2-minute timeout, if the "duplicate" host is over 15
A-records from the last time you tried, you need to retry...

A more subtle hole I've seen MTA's fall into - do the MX and A record lookups,
get a list of 20 or so A records with a 30-min TTL.  Start trying, 2 minute
timeout on each.  When you get to the 16th and fail to notice it's stale and
needs to be looked up again.

Figuring out what to try out of the NEW list of 20, which may or may not
include the same A records as the first time, or partial overlap, and some that
you've tried at the beginning and are set for retry, and some that are recent,
and some you've expired and haven't tried, and some that are new, is left as an
exercise for the masochist  A case could be made that if you've been trying so
long that the TTL's have expired, it's time to Just Give Up, queue it, and go
have a beer or something and try later, even though you have untried addresses.

The usual non-standard-compliant way to work around this is to only allow a
short timeout (5-10 seconds or so) for the *FIRST* try at the TCP 3-packet
handshake, and if that fails, retry after 30 mins or so.  Combining this with a
MTA-wide "recent status" table allows you to fairly quickly detect that once
again, AOL/Yahoo/Hotmail/whoever has gone belly up and not strangle yourself on
retries (if ONE piece of mail couldn't reach any Hotmail server, the next piece
2.5 minutes later probably can't either, so you're better off not even trying
to deliver the second, and stashing both of them in a queue and retry them both
in 30 mins...)

> > If the Protocol Police are going to write a ticket for this one,
> > we need to clean it up so the Protocol Lawyers and Protocol Judge
> > don't spend all afternoon arguing. ;)

> I hope I didn't open a can of worms or add more work for anyone :-)

If we didn't want this work, we'd not be subscribed  - this is a 100% volunteer
effort, except for the IETF Secretariat who are not on this list anyhow. :)

--==_Exmh_1106085364P
Content-Type: application/pgp-signature

--==_Exmh_1106085364P--

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
Date: Sun, 4 Jan 2004 15:32:09 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "IETF-SMTP" <[email-masked]>
Sent: Sunday, January 04, 2004 2:09 PM
Subject: Re: RFC 2821 Address Resolution

> Remember that round-robin is for the *server cluster*'s
> convenience, not the client's. It's so that *overall*, clients
> are evenly distributed across the round-robin list and don't all
> pile on the first one.

So an inquiring client should "randomize" it (again) to  minimize the
possibility that two or more current client snapshots are not going to hit
the same servers at the same time?

> In fact, a single client can't even tell round-robin is in use
> unless you ask several times (I once had the luck of doing a 'dig'
> of a address I *knew* was round-robined from the server listed
> in the NS (so any caching at my end didn't matter), and I had to
> ask *five times* before I got a different list. I just happened to
> be the N'th querie 4 times in a row, so it had round-robined back
> to where I was)...

This was what was happening with gethostbyname(). A direct A record lookup
yield a round-robin result, but the Windows socket function resolved it from
its cache first returning the same address over and over again until it
timed out and fetched a new set of records.

That is when I realized that the MX query "may" return the A records so this
should be the opportunity to resolve the address if available.  No need for
the gethostbyname() call in this case.

> Yes, that's sub-optimal if you cached a list that has the downed
> host first in a list of 4, but hopefully only 1/4 of the hosts
> end up in that situation.

For aol.com, I'm noticing it is usually 2 of the 4 that are down (no A
record in the MX query) with round-robin results.

> As I said, not guaranteed.  Your need to look these A records up
> as well.

> You are allowed to optimize and not look them up *if* the A-less
> MX'x are secondaries and as a result may not actually get
> referenced.  Wait till you need them.  If you're really clever,
> you'll fire the lookup when your original list is "getting low",
> so the DNS lookups happen in parallel with the last few tries of
> A records, balancing "don't make un-needed queries unless you
> have to" with "don't stall waiting for queries".

Ha! This is the exact "clever" logic I am currently testing out! :-)  Seems
to produce a legitimate final list.  This worked with the test domains like
AOL.COM that happen to have equal preference.

But what do you do with multi-MX where one or more is down and have lower
preference numbers than the rest?

> RFC2821, section 5, says: ..........

> So you're required to exhaust your list of mx1/mx2 A records
> first, then try the mx4 A records.  You're required to randomize
> whether you try mx1 or mx2 first.  However, the spec does *not*
> say what to do if you have multiple A records from multiple MX:

> 1) It's unclear if you should try the first mx1, the first mx2,
> the second mx1, the second mx2, or first try all the mx1 and then
> all the mx2.

Group them and then randomized?

> 2) It's unclear if you encounter a duplicate address you've tried
> already on ONE mx if you should remove it from the list, or
> re-try if it's the other MX.

I'm removing it from the list currently.  If the server is not responding,
why try it again during the current send mail
session?

> If the Protocol Police are going to write a ticket for this one,
> we need to clean it up so the Protocol Lawyers and Protocol Judge
> don't spend all afternoon arguing. ;)

I hope I didn't open a can of worms or add more work for anyone :-)

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Date: Sun, 04 Jan 2004 15:23:55 -0500
From: "Richard O. Hammer" <[email-masked]>
To: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution
Sender: [email-masked]

Hector,

I am a newcomer to DNS so you must consider anything I write to be 
Non-authoritative.  But last month I contributed code to the 
james.apache.org project which attempts to deal correctly with 
multi-homed MX hosts, and that code seems to have survived so far. 
The earlier code in James had found only one IP address for each MX 
host, even if that host had multiple homes (multiple A records), just 
as you describe for your former situation.

Just to make sure we are using the same language, a "host" is a name 
like mailin-02.mx.aol.com which you get in an MX record.  A "home" is 
a single IP address (maybe even a single computer, but this is 
complicated enough already) which you get in an A record, assuming of 
course that I know what I am talking about, on which question I would 
bet even odds.

Oh yeah, and a "domain" is a thing like "aol.com".  A domain may have 
multiple hosts, each with a priority number.  A host may have multiple 
homes, with no indication of priority among the homes.  To establish 
an SMTP connection to a domain, you should ideally not give up until 
you have tried all the homes of all the hosts.

As I understand RFC 2821, you are supposed to randomize any set of 
hosts which have the same priority, but you are not asked to randomize 
the multiple homes of any given host.

Thanks for introducing me to the nslookup command in Windows.

Hector Santos wrote:
 > The logic I used is:
 >
 > 1) Sort the MX records (low to high) preference,
 > 2) For each MX, add the A records to the final list removing 
duplicate ips.

That sounds right to me.  My code does not look for or remove 
duplicate IPs; I do not remember seeing this suggested in the RFC, but 
it makes sense.

 > what should I do with?
 >
 > aol.com MX preference = 15, mail exchanger = mailin-03.mx.aol.com
 > aol.com MX preference = 15, mail exchanger = mailin-04.mx.aol.com
 >
 > where there is no A records for these exchanged returned in the MX 
query?

I think the A records appended to the reply to your MX query are put 
there only as an expedient, a way to fill up a packet with information 
which may be useful.  I think a DNS resolver is supposed to cache this 
A-record information, since it may save a later call to a DNS server. 
  Further, the list of A-records appended to a MX query does not 
pretend to be complete.  The lists which you show appear to me to be 
truncated, perhaps at a size limit determined by the packet size.

I think you should make more calls to DNS, asking for A records.  My 
Java code calls the DNS resolver again with an A-record query for each 
host.  And here, with my newly learned nslookup command, I note that I 
can get this (seven IP addresses to add to the list:

C:\j>nslookup -querytype=a mailin-03.mx.aol.com
Server:  dsldns-nc.sprint-hsd.net
Address:  63.162.197.69

Non-authoritative answer:
Name:    mailin-03.mx.aol.com
Addresses:  64.12.138.57, 64.12.138.120, 205.188.156.249, 205.188.157.25
           205.188.159.217, 64.12.137.121, 64.12.137.152

 > Anyway, what would you think the final list should be?

Your lists look correct to me, as far as prioritization of hosts is 
concerned, although perhaps incomplete as there may be many more 
A-records to be found.

An optimization: Since there are potentially many homes (maybe 
sometimes even 100) for a given domain, and since my logs indicate to 
me that the first attempt to connect to any home usually succeeds, 80% 
of the time or better, it may be wasteful to prepare the entire list 
of homes, sorted by host-priority, before attempting to connect to the 
first home in the list.  The first attempt will probably succeed and 
make the rest of the list useless.  Initially, my code does no more 
DNS lookups than necessary to find the first home that it should 
attempt for a domain; it postpones further DNS lookups and sorting 
until needed, if ever.

Rich Hammer
Hillsborough, N.C.

To: [email-masked] (Jun-ichiro itojun Hagino)
Cc: [email-masked]
Subject: Re: spam relays on secondary MX 
From: [email-masked]
Date: Sun, 04 Jan 2004 15:09:52 -0500
Sender: [email-masked]

--==_Exmh_813694480P
Content-Type: text/plain; charset=us-ascii

On Mon, 05 Jan 2004 04:28:26 +0900, [email-masked] (Jun-ichiro itojun Hagino)  said:
> 	there seem to be many interesting discussions on secondary MX and spam
> 	relays.  basically, secondary MXes normally do not check the existence
> 	of accounts (like [email-masked]) and it will get queued, while primary
> 	MX can reject it at instant.  this may be ASRG topic, but i guess
> 	there's something to be done in SMTP level.

The only reason this is done by the spammers at all is because often in
these situations, the primary MX will blindly accept mail from the secondary
without applying anti-spam controls.  Certainly, such broken sites are in
the minority - but when 80% of your attempts get blocked, finding a loophole
that exists even in 2-3% of sites means an 10% efficiency gain....

I can't recommend any change in SMTP to counter this - the same sites that
have the problem are the same sites that are unlikely to deploy any changes to
fix the problem.

--==_Exmh_813694480P
Content-Type: application/pgp-signature

--==_Exmh_813694480P--

To: [email-masked]
Subject: spam relays on secondary MX
Date: Mon,  5 Jan 2004 04:28:26 +0900 (JST)
From: [email-masked] (Jun-ichiro itojun Hagino)
Sender: [email-masked]

	there seem to be many interesting discussions on secondary MX and spam
	relays.  basically, secondary MXes normally do not check the existence
	of accounts (like [email-masked]) and it will get queued, while primary
	MX can reject it at instant.  this may be ASRG topic, but i guess
	there's something to be done in SMTP level.

	some mailing list (do not remember which) even showed the attack ratio
	against secondary MX and primary MX, and secondary MX is attacked
	much frequently.

	http://www.google.com/search?q=mx+secondary+spam&sourceid=mozilla-search&start=0&start=0&ie=utf-8&oe=utf-8

itojun

To: Hector Santos <[email-masked]>
Cc: IETF-SMTP <[email-masked]>
Subject: Re: RFC 2821 Address Resolution 
From: [email-masked]
Date: Sun, 04 Jan 2004 14:09:18 -0500
Sender: [email-masked]

--==_Exmh_572938684P
Content-Type: text/plain; charset=us-ascii

On Sun, 04 Jan 2004 07:40:33 EST, Hector Santos <[email-masked]>  said:

(For the list - Another few protocol nit mentioned later on - I seem to be noticing
them this week :)

> I ask because I have learned a few things lately about the current design of
> our DNS MX resolver resulting from an initial investigation why Microsoft
> Window socket function gethostbyname() returned the same IP address for a
> single MX, multi-home (multiple A records) domain.  The DNS A record lookup
> shows a load balancing/sharing (rotation of the A records), but
> gethostbyname() seems to return its internal cache for the single MX
> exchange name lookup.

Remember that round-robin is for the *server cluster*'s convenience, not the
client's. It's so that *overall*, clients are evenly distributed across the
round-robin list and don't all pile on the first one.  In fact, a single client
can't even tell round-robin is in use unless you ask several times (I once had
the luck of doing a 'dig' of a address I *knew* was round-robined from the
server listed in the NS (so any caching at my end didn't matter), and I had to
ask *five times* before I got a different list. I just happened to be the N'th
querie 4 times in a row, so it had round-robined back to where I was)...

So that's actually permissible, as long as it obeys the TTL on the cached entry
and re-resolves it if it gets stale.  Although any single host that does
caching doesn't get the benefit of the round-robin, that's OK because an
application SHOULD (rfc sense) try the A records one at a time.  Yes, that's
sub-optimal if you cached a list that has the downed host first in a list of 4,
but hopefully only 1/4 of the hosts end up in that situation.

> Well, looking at how DNS resolves an MX lookup, our design to use
> gethostbyname() seems to be incorrect and redundant since the A record(s)
> may be returned with the MX query.

However, it's not guaranteed.  As you notice below...

> what should I do with?
> 
> aol.com MX preference = 15, mail exchanger = mailin-03.mx.aol.com
> aol.com MX preference = 15, mail exchanger = mailin-04.mx.aol.com
> 
> where there is no A records for these exchanged returned in the MX query?

As I said, not guaranteed.  Your need to look these A records up as well.

You are allowed to optimize and not look them up *if* the A-less MX'x are
secondaries and as a result may not actually get referenced.  Wait till you
need them.  If you're really clever, you'll fire the lookup when your original
list is "getting low", so the DNS lookups happen in parallel with the last few
tries of A records, balancing "don't make un-needed queries unless you have to"
with "don't stall waiting for queries".

> Similarly for yahoo.com the final list would be based on my new lookup
> logic:
> 
> mx2.mail.yahoo.com      internet address = 64.157.4.78
> mx2.mail.yahoo.com      internet address = 64.156.215.5
> mx2.mail.yahoo.com      internet address = 64.156.215.6
> mx1.mail.yahoo.com      internet address = 64.157.4.79
> mx1.mail.yahoo.com      internet address = 64.156.215.7
> mx4.mail.yahoo.com      internet address = 66.218.86.156
> mx4.mail.yahoo.com      internet address = 66.218.86.253
> mx4.mail.yahoo.com      internet address = 66.218.86.254
> mx4.mail.yahoo.com      internet address = 67.28.114.32
> mx4.mail.yahoo.com      internet address = 216.155.197.63

RFC2821, section 5, says:

   Multiple MX records contain a preference indication that MUST be used
   in sorting (see below).  Lower numbers are more preferred than higher
   ones.  If there are multiple destinations with the same preference
   and there is no clear reason to favor one (e.g., by recognition of an
   easily-reached address), then the sender-SMTP MUST randomize them to
   spread the load across multiple mail exchangers for a specific
   organization.

   The destination host (perhaps taken from the preferred MX record) may
   be multihomed, in which case the domain name resolver will return a
   list of alternative IP addresses.  It is the responsibility of the
   domain name resolver interface to have ordered this list by
   decreasing preference if necessary, and SMTP MUST try them in the
   order presented.

So you're required to exhaust your list of mx1/mx2 A records first, then
try the mx4 A records.  You're required to randomize whether you try mx1
or mx2 first.  However, the spec does *not* say what to do if you have
multiple A records from multiple MX:

1) It's unclear if you should try the first mx1, the first mx2, the second mx1,
the second mx2, or first try all the mx1 and then all the mx2.

2) It's unclear if you encounter a duplicate address you've tried already on ONE mx
if you should remove it from the list, or re-try if it's the other MX.

If the Protocol Police are going to write a ticket for this one, we need to clean
it up so the Protocol Lawyers and Protocol Judge don't spend all afternoon
arguing. ;)

--==_Exmh_572938684P
Content-Type: application/pgp-signature

--==_Exmh_572938684P--

From: "Hector Santos" <[email-masked]>
To: "IETF-SMTP" <[email-masked]>
Subject: RFC 2821 Address Resolution
Date: Sun, 4 Jan 2004 07:40:33 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

I have a question about MX record lookups.  The RFC discusses about
expanding or creating the list of IP addresses that a SMTP send mail session
will attempt.   I would like to see what other smtp developers will do with
the MX lookup, for example for AOL.COM,  to generate the final
sorted/ordered list of IPs:

Here is a current snap shot of a NSLOOKUP MX lookup for aol.com and for
yahoo.com

D:\local\wc5\wcsmtp\wcdns>nslookup -querytype=mx aol.com
Server:  ns.bct.bellsouth.net
Address:  205.152.110.252

Non-authoritative answer:
aol.com MX preference = 15, mail exchanger = mailin-02.mx.aol.com
aol.com MX preference = 15, mail exchanger = mailin-03.mx.aol.com
aol.com MX preference = 15, mail exchanger = mailin-04.mx.aol.com
aol.com MX preference = 15, mail exchanger = mailin-01.mx.aol.com

aol.com nameserver = dns-07.ns.aol.com
aol.com nameserver = dns-01.ns.aol.com
aol.com nameserver = dns-02.ns.aol.com
aol.com nameserver = dns-06.ns.aol.com
mailin-01.mx.aol.com    internet address = 64.12.137.89
mailin-01.mx.aol.com    internet address = 64.12.137.184
mailin-01.mx.aol.com    internet address = 64.12.138.57
mailin-01.mx.aol.com    internet address = 64.12.138.152
mailin-01.mx.aol.com    internet address = 205.188.156.185
mailin-01.mx.aol.com    internet address = 205.188.158.121
mailin-01.mx.aol.com    internet address = 205.188.159.57
mailin-02.mx.aol.com    internet address = 205.188.158.121
mailin-02.mx.aol.com    internet address = 205.188.159.57
mailin-02.mx.aol.com    internet address = 64.12.137.89
mailin-02.mx.aol.com    internet address = 64.12.137.184
mailin-02.mx.aol.com    internet address = 64.12.138.89
mailin-02.mx.aol.com    internet address = 64.12.138.120
mailin-02.mx.aol.com    internet address = 205.188.156.185

D:\local\wc5\wcsmtp\wcdns>nslookup -querytype=mx yahoo.com
Server:  ns.bct.bellsouth.net
Address:  205.152.110.252

Non-authoritative answer:
yahoo.com       MX preference = 1, mail exchanger = mx2.mail.yahoo.com
yahoo.com       MX preference = 5, mail exchanger = mx4.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = mx1.mail.yahoo.com

yahoo.com       nameserver = ns1.yahoo.com
yahoo.com       nameserver = ns2.yahoo.com
yahoo.com       nameserver = ns3.yahoo.com
yahoo.com       nameserver = ns4.yahoo.com
yahoo.com       nameserver = ns5.yahoo.com
mx1.mail.yahoo.com      internet address = 64.157.4.79
mx1.mail.yahoo.com      internet address = 64.156.215.5
mx1.mail.yahoo.com      internet address = 64.156.215.6
mx1.mail.yahoo.com      internet address = 64.156.215.7
mx1.mail.yahoo.com      internet address = 64.157.4.78
mx2.mail.yahoo.com      internet address = 64.157.4.78
mx2.mail.yahoo.com      internet address = 64.156.215.5
mx2.mail.yahoo.com      internet address = 64.156.215.6
mx4.mail.yahoo.com      internet address = 66.218.86.156
mx4.mail.yahoo.com      internet address = 66.218.86.253
mx4.mail.yahoo.com      internet address = 66.218.86.254
mx4.mail.yahoo.com      internet address = 67.28.114.32
mx4.mail.yahoo.com      internet address = 216.155.197.63

Based on each snap shot, what would you generate (and why) for total
possible IP list, disregarding system policy options a server might offer to
limit the total servers to try?

I ask because I have learned a few things lately about the current design of
our DNS MX resolver resulting from an initial investigation why Microsoft
Window socket function gethostbyname() returned the same IP address for a
single MX, multi-home (multiple A records) domain.  The DNS A record lookup
shows a load balancing/sharing (rotation of the A records), but
gethostbyname() seems to return its internal cache for the single MX
exchange name lookup.

Well, looking at how DNS resolves an MX lookup, our design to use
gethostbyname() seems to be incorrect and redundant since the A record(s)
may be returned with the MX query.

Also, I noticed the current design of the our resolver is to return the
total servers based on the number of MX records.  So if only 1 MX record is
turned for a multiple A records system,  only 1 attempt is made.   That
seems to be incorrect according to the specs.

I seem I have worked it out according to the specs.  Now I need to verify
the results our new MX resolver.

The logic I used is:

1) Sort the MX records (low to high) preference,
2) For each MX, add the A records to the final list removing duplicate ips.

For aol.com, the final list would be:

mailin-02.mx.aol.com    internet address = 205.188.158.121
mailin-02.mx.aol.com    internet address = 205.188.159.57
mailin-02.mx.aol.com    internet address = 64.12.137.89
mailin-02.mx.aol.com    internet address = 64.12.137.184
mailin-02.mx.aol.com    internet address = 64.12.138.89
mailin-02.mx.aol.com    internet address = 64.12.138.120
mailin-02.mx.aol.com    internet address = 205.188.156.185
mailin-01.mx.aol.com    internet address = 64.12.138.57
mailin-01.mx.aol.com    internet address = 64.12.138.152

what should I do with?

aol.com MX preference = 15, mail exchanger = mailin-03.mx.aol.com
aol.com MX preference = 15, mail exchanger = mailin-04.mx.aol.com

where there is no A records for these exchanged returned in the MX query?

In our current version,  the current logic simply grabs the 4 MX records and
does a gethostbyname() to get the IP.   This didn't solve the load
balancing, but more importantly,  it does not seem to be correct according
to the RFC.  The potential list of servers to try is much higher.

Similarly for yahoo.com the final list would be based on my new lookup
logic:

mx2.mail.yahoo.com      internet address = 64.157.4.78
mx2.mail.yahoo.com      internet address = 64.156.215.5
mx2.mail.yahoo.com      internet address = 64.156.215.6
mx1.mail.yahoo.com      internet address = 64.157.4.79
mx1.mail.yahoo.com      internet address = 64.156.215.7
mx4.mail.yahoo.com      internet address = 66.218.86.156
mx4.mail.yahoo.com      internet address = 66.218.86.253
mx4.mail.yahoo.com      internet address = 66.218.86.254
mx4.mail.yahoo.com      internet address = 67.28.114.32
mx4.mail.yahoo.com      internet address = 216.155.197.63

and finally again, for our current logic, the 3 MX records would be used for
the final list of servers for yahoo.com.

Anyway, what would you think the final list should be?

Thanks in advance for any input you may have on this.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: what to say on timeout?
Date: Sat, 3 Jan 2004 23:03:38 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Richard,

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, January 03, 2004 9:58 PM
Subject: Re: what to say on timeout?

>
> CORRECTION: I meant to refer to RFC 1939, not 1938.
>
> Richard O. Hammer wrote:
> >
> > I have looked in RFC 2821 but not yet found any explicit text which
> > tells what to do in this case.  But, for a comparison, RFC 1938 (dealing
> > with POP3) Section 3 says:
> >
> >    A POP3 server MAY have an inactivity autologout timer.  ...
> >   When the timer expires, the session does NOT enter
> >    the UPDATE state--the server should close the TCP connection without
> >    removing any messages or sending any response to the client.

hmmm,  I guess I can see where one can get the idea that you can send a
unsolicited state machine response to a client, but  this has to do with the
successful completion and acknowledgement of a POP3 download session to
trigger the POP3 server logic to perform the final mail database updates
(i.e., mark downloaded mail as received).

In other words,  you only want the server to perform the UPDATE when the
QUIT command is sent by the client.  (RFC 1939, section 3, para 4)

If the QUIT command is not sent by the client, the SERVER should not perform
any mail database update because it could mean there was an unexpected drop
in the connection.   That should include an inactivity timeout.

No QUIT command, no update.   If I recall one client, an old version of the
Microsoft Outlook for the MAC that would drop the connection after the
download.   This would cause a problem with users who didn't complete the
download and tries again.  MS was quickly made aware of the RFC spec bug and
an update was provided.

Hope this helps.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: what to say on timeout?
Date: Sat, 3 Jan 2004 22:34:05 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

Richard,

SMTP is a client/server state machine protocol .   Like in most state
machines,  the SMTP server should not send out a response or information
unless it received a request, command or signal (as in the SMTP connection
signal) from the client. SMTP clients are not designed to received
unsolicited responses.  Simply log the timeout and drop the connection.

Hope this helps.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

----- Original Message ----- 
From: "Richard O. Hammer" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, January 03, 2004 8:35 PM
Subject: what to say on timeout?

>
> When a SMTP server breaks a connection to a SMTP client because of
> long inactivity (as suggested in RFC 2821 Section 4.5.3.2), should it
> break the connection silently? or should it first send a 451, 554, or
> some other reply code?
>
> Thanks,
> Rich Hammer
>
>

Date: Sat, 03 Jan 2004 21:58:57 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

CORRECTION: I meant to refer to RFC 1939, not 1938.

Richard O. Hammer wrote:
> 
> I have looked in RFC 2821 but not yet found any explicit text which 
> tells what to do in this case.  But, for a comparison, RFC 1938 (dealing 
> with POP3) Section 3 says:
> 
>    A POP3 server MAY have an inactivity autologout timer.  ...
>   When the timer expires, the session does NOT enter
>    the UPDATE state--the server should close the TCP connection without
>    removing any messages or sending any response to the client.

Date: Sat, 03 Jan 2004 21:39:09 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

I have looked in RFC 2821 but not yet found any explicit text which 
tells what to do in this case.  But, for a comparison, RFC 1938 
(dealing with POP3) Section 3 says:

    A POP3 server MAY have an inactivity autologout timer.  ...
   When the timer expires, the session does NOT enter
    the UPDATE state--the server should close the TCP connection without
    removing any messages or sending any response to the client.

Rich

Date: Sat, 03 Jan 2004 21:23:24 -0500
From: John C Klensin <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>, [email-masked]
Subject: Re: what to say on timeout?
Sender: [email-masked]

--On Saturday, 03 January, 2004 20:35 -0500 "Richard O. Hammer" 
<[email-masked]> wrote:

>
> When a SMTP server breaks a connection to a SMTP client
> because of long inactivity (as suggested in RFC 2821 Section
> 4.5.3.2), should it break the connection silently? or should
> it first send a 451, 554, or some other reply code?

Rich, I don't have time to look up the text (perhaps someone 
else does), but the general theory is that an SMTP server should 
never silently break connections but should send back a reply 
code and then wait for the client to issue a QUIT.

There is an exception of sorts, which should probably be more 
explicitly documented since it covers most of the real cases 
(people who agree on more explicitness should send text).  Since 
SMTP is defined as essentially synchronous, there is no 
mechanism for a server which has already sent a reply code and 
is now waiting for a command to send a reply code of any sort. 
So it has little choice but to give up and close the connection.

The standard permits some flexibility about these cases 
(especially the first one) when driven by operational 
necessities: in addition to all of the "what to do about the 
spammers" cases/ discussions, spending a lot of time waiting for 
that QUIT when you've already timed out waiting for a command 
just doesn't make much sense.  But, if you are in a position to 
send a reply code (i.e., you haven't already done so and are 
waiting for a command, if that case actually exists), I would be 
inclined to send it, wait at least a short time for a reply, and 
then give up, rather than giving up without sending the code.

      john

Date: Sat, 03 Jan 2004 20:35:08 -0500
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: what to say on timeout?
Sender: [email-masked]

When a SMTP server breaks a connection to a SMTP client because of 
long inactivity (as suggested in RFC 2821 Section 4.5.3.2), should it 
break the connection silently? or should it first send a 451, 554, or 
some other reply code?

Thanks,
Rich Hammer

Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
From: "B. Johannessen" <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: [email-masked]
Organization: http://db.org/
Date: Sat, 03 Jan 2004 16:49:38 +0000
Sender: [email-masked]

On Sat, 2004-01-03 at 15:25, John C Klensin wrote:
> "B. Johannessen" <[email-masked]> wrote:
> > - the syntax and possible values of parameters associated with
> >   the EHLO keyword value;
> 
> Better phrasing would be welcomed, but that really refers to the 
> use of the permitted extensions in other commands.

The phrasing is just fine. The problem was my reading. Somehow
I must have substituted "command/verb" for "keyword value" when
I parsed that sentence. My fault entirely.

> > cases, to allow parameters for EHLO? Or to put it another way,
> > was there any good reasons *not* to allow parameters for EHLO?
> 
> ...and no one could come up with a good reason 
> for such parameters....

I was thinking that it may somehow be used to save a step in the
dialogue. Now that I think about it though, I can't really come
up with a useful example. At least not without turning the entire
negotiation process up-side-down.

> ...More generally, one really doesn't want to 
> have multiple negotiation models floating around...

Sounds like a good enough reason to me.

Thanks for clearing this up and providing some insight into /why/.

	Bob

-- 
-=[ B. Johannessen | [email-masked] -=- http://db.org/ | +4797152009 ]=-
-=[ Mail & Spam - News, Drafts & Standards - http://db.org/blog/ ]=-

Date: Sat, 03 Jan 2004 10:25:43 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
Sender: [email-masked]

My apologies to all -- I was distracted yesterday evening and 
should not try to write ABNF in that state.  Trying to put this 
optional string into "Domain" would, indeed, mess up all sorts 
of things and was not intended.  Instead, the correct syntax 
changes should, I think, be (in 4.1.1.1)

      ehlo   = "EHLO" SP ( Domain / Explained-literal ) CRLF
      helo   = "HELO" SP Domain CRLF

and then in 4.1.1.1 or 4.1.2, add
      Explained-literal = address-literal [string]

i.e., for backward compatibility (and some stupidity and spam 
resistance) HELO does not change at all, no embedded spaces are 
permitted after "HELO ".   EHLO either gets a strict domain name 
string, or a literal, either with or without explanatory text.

Of course, if the better solution were to permit the explanatory 
text with the domain name as well (I'm convinced this morning, 
after skimming the thread, that it is not desirable), then the 
transformation above to "Explained-domain" is obvious.  And HELO 
(and other commands) still don't change.

Also...

--On Friday, 02 January, 2004 23:41 +0000 "B. Johannessen" 
<[email-masked]> wrote:

> For
> some reason I had the idea that (SMTP) extensions where allowed
> to define parameters for EHLO as well. Must have been the 3rd
> item in the requirements that got me confused:
>
> - the syntax and possible values of parameters associated with
>   the EHLO keyword value;

Better phrasing would be welcomed, but that really refers to the 
use of the permitted extensions in other commands.

> But on the other hand (and probably part of the reason I got it
> wrong in the first place) wouldn't it have been useful, in some
> cases, to allow parameters for EHLO? Or to put it another way,
> was there any good reasons *not* to allow parameters for EHLO?

If I remember the reasoning at the time, it was largely that 
unnecessary flexibility just leads to problems and 
incompatibilities and no one could come up with a good reason 
for such parameters.  More generally, one really doesn't want to 
have multiple negotiation models floating around.  The current 
one is "client tells server it is willing to accept capabilities 
for negotiation, server tells client what capabilities it 
supports, client asks for whatever options with that range that 
it wants".  The only value for an EHLO parameter that I can see 
would be to introduce a "client tells server which options it 
wants to negotiate, server indicates what it is willing to 
accept from that range, client asks for a subset of the agreed 
list".  Mostly just different; the only major advantage of the 
current model is that it doesn't get tied up with requiring a 
longer line length on EHLO (commands or parameters authorized in 
the EHLO response can expand the required minimum command 
length, and continuation lines bypass the limit for responses, 
but there is no way to do it for the EHLO command line).

    john

To: Hector Santos <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO 
From: [email-masked]
Date: Fri, 02 Jan 2004 19:39:00 -0500
Sender: [email-masked]

--==_Exmh_-1156566864P
Content-Type: text/plain; charset=us-ascii

On Fri, 02 Jan 2004 18:55:54 EST, Hector Santos <[email-masked]>  said:

> Since it is only meant to help in the logging/tracking on the helo/ehlo
> domain literal and for maximum backward compatibility,  do not have the
> [string] element in the domain ABNF, and If possible, to be clear and
> specific, add a new  "machine-domain" ABNF:
> 
>       helo            = "HELO" SP Domain CRLF
>       ehlo            = "EHLO" SP Machine-Domain CRLF
> 
>       Machine-Domain = domain [ string ]
> 
> This way, there would be no confusion with other ABNF specifications
> requiring the domain element.

D'Oh! -- H. Simpson.

Yes, you are totally right that this sort of change is preferred.  As written,
machine-domain allows text after the FQDN on the EHLO, but I'm not sure
it's worth quibbling about. :)

--==_Exmh_-1156566864P
Content-Type: application/pgp-signature

--==_Exmh_-1156566864P--

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO 
Date: Fri, 2 Jan 2004 19:23:21 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: "B. Johannessen" <[email-masked]>; <[email-masked]>
Sent: Friday, January 02, 2004 6:14 PM
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO

> A separate issue comes to mind - a number of systems reject the
> format mentioned in 4.1.1.1.  In particular, Sendmail 8.12.4 included
> this change:
>
>         If AllowBogusHELO is set to false (default) then also complain if
>                 the argument to HELO/EHLO contains white space.  Suggested
>                 by Seva Gluschenko of Cronyx Plus.
>
> (Which incidentally was how I opened this can of worms to begin with).

Ahhhhhh, good point.  Ironically,  I just added this optional SPACE check
ourselves last month to our beta ware based on seeing it in some captured
logs. example:

HELO [filename /home/admin/domains.txt]
250 winserver.com, Pleased to meet you.
MAIL FROM: <GHaF@[filename /home/admin/domains.txt]>
552 malformed non-null return path: <GHaF@[filename

I honestly didn't see the specs allows for a string element in a bracketed
domain literal.  In this example, it is bad syntax, but I can see how this
can present issues.

> Fortunately, at least Sendmail allows retrying with a less-weird EHLO, so
> there's a way for future systems to drop back.  Unless of course some
system
> out there gets annoyed at seeing a second EHLO instead of a HELO
drop-back.
> Isn't backward-combatability fun?

I have seen this myself, but what I have seen is "spammers" reissuing 2-3
HELO commands when presended with a:

                "55x illegal/invalid helo/ehlo syntax/domain

A few months ago, I added an option to check for HELO/EHLO syntax checking
which essentially addresses local domain spoofing,  spoofed domain literals
and illegal (non-bracketed) domain literals.

Example:  Connection IP:  X.Y.Z.W

           helo   [xxx.]yourdomain.com

if yourdomain.com is yours,  it is checked again the connection ip address
using the proposed IETF draft DMP specification.

          helo [A.B.C.D]

A.B.C.D  must be equal to X.Y.Z.W

          helo A.B.C.D

illegal syntax - no brackets (AllowNoBrackets option provided)

          helo domain string

illegal syntax - illegal character or white space found.

In 3 months worth of statistics,  on average 15% of sessions will be blocked
because of the above.  Some systems will reissue 2-3 helo commands:

    C:  helo [A.B.C.D]
    S:  501 invalid HELO address  (because it doesn't match the connect ip)
    C:  helo A.B.C.D
    S:  501 invalid HELO address   (because of no brackets)

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

From: "Hector Santos" <[email-masked]>
To: <[email-masked]>
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
Date: Fri, 2 Jan 2004 18:55:54 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>; "B. Johannessen" <[email-masked]>;
<[email-masked]>
Sent: Friday, January 02, 2004 5:41 PM
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO

>
> Is that what is wanted, or should the ABNF be
>
>        Domain = ( (sub-domain 1*("." sub-domain)) /
>           address-literal ) [ string ]
>

Will this not conflict with other ABNFs requiring an domain specification,
such as MAIL FROM?

      "MAIL FROM:" ("<>" / Reverse-Path)
                       [SP Mail-parameters] CRLF

      Reverse-path = Path
      Forward-path = Path
      Path = "<" [ A-d-l ":" ] Mailbox ">"
      A-d-l = At-domain *( "," A-d-l )
            ; Note that this form, the so-called "source route",
            ; MUST BE accepted, SHOULD NOT be generated, and SHOULD be
            ; ignored.
      At-domain = "@" domain

examples:

      mail from: <[email-masked]>
      mail from: <joe@[X.X.X.X] see contact xyx>

Syntaxtically, it is correct but I question which software expects something
like this to perform the proper parsing. Honestly, this is something we
would have to change in our own server to honor the optional [string] in a
possible domain literal reverse-path.

My input would be:

Since it is only meant to help in the logging/tracking on the helo/ehlo
domain literal and for maximum backward compatibility,  do not have the
[string] element in the domain ABNF, and If possible, to be clear and
specific, add a new  "machine-domain" ABNF:

      helo            = "HELO" SP Domain CRLF
      ehlo            = "EHLO" SP Machine-Domain CRLF

      Machine-Domain = domain [ string ]

This way, there would be no confusion with other ABNF specifications
requiring the domain element.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
From: "B. Johannessen" <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: [email-masked], [email-masked]
Organization: http://db.org/
Date: Fri, 02 Jan 2004 23:41:09 +0000
Sender: [email-masked]

On Fri, 2004-01-02 at 22:41, John C Klensin wrote:
> "B. Johannessen" <[email-masked]> wrote:
> > I'd say it's section 4.1.1.1 that's b0rked. If you allow free
> > form "information" after the FQDN/address-literal it makes it
> > impossible to parse (ESMTP) parameters for EHLO.
[clip]
> capability list makes sense.  There has never been any model for 
> the client to announce anything to the server at EHLO time, nor 
> to put any ESMTP parameters there.

Just re-read the relevant part of 2821 (2.2.2 Definition and
Registration of Extensions), and you are of course correct. For
some reason I had the idea that (SMTP) extensions where allowed
to define parameters for EHLO as well. Must have been the 3rd
item in the requirements that got me confused:

- the syntax and possible values of parameters associated with
  the EHLO keyword value;

But on the other hand (and probably part of the reason I got it
wrong in the first place) wouldn't it have been useful, in some
cases, to allow parameters for EHLO? Or to put it another way,
was there any good reasons *not* to allow parameters for EHLO?

	Bob

-- 
-=[ B. Johannessen | [email-masked] -=- http://db.org/ | +4797152009 ]=-
-=[ Mail & Spam - News, Drafts & Standards - http://db.org/blog/ ]=-

To: John C Klensin <[email-masked]>
Cc: "B. Johannessen" <[email-masked]>, [email-masked]
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO 
From: [email-masked]
Date: Fri, 02 Jan 2004 18:14:17 -0500
Sender: [email-masked]

--==_Exmh_-1369926300P
Content-Type: text/plain; charset=us-ascii

On Fri, 02 Jan 2004 17:41:26 EST, you said:
> First, the ABNF is wrong given the text and, on behalf of myself 
> and the several people who checked it, my/our apologies.

A separate issue comes to mind - a number of systems reject the
format mentioned in 4.1.1.1.  In particular, Sendmail 8.12.4 included
this change:

        If AllowBogusHELO is set to false (default) then also complain if
                the argument to HELO/EHLO contains white space.  Suggested
                by Seva Gluschenko of Cronyx Plus.

Such sendmails reject it:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 turing-police.cc.vt.edu ESMTP Sendmail 8.13.0.PreAlpha5/8.13.0.PreAlpha5; Fri, 2 Jan 2004 18:08:03 -0500
ehlo [127.0.01.1] test.domain.name
501 5.0.0 Invalid domain name

(Which incidentally was how I opened this can of worms to begin with).

Fortunately, at least Sendmail allows retrying with a less-weird EHLO, so
there's a way for future systems to drop back.  Unless of course some system
out there gets annoyed at seeing a second EHLO instead of a HELO drop-back.

Isn't backward-combatability fun?

--==_Exmh_-1369926300P
Content-Type: application/pgp-signature

--==_Exmh_-1369926300P--

To: John C Klensin <[email-masked]>
Cc: "B. Johannessen" <[email-masked]>, [email-masked]
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO 
From: [email-masked]
Date: Fri, 02 Jan 2004 17:58:22 -0500
Sender: [email-masked]

--==_Exmh_-1465556040P
Content-Type: text/plain; charset=us-ascii

On Fri, 02 Jan 2004 17:41:26 EST, you said:
> The text, as written, implies that the ABNF should have been
>
>       Domain = (sub-domain 1*("." sub-domain)) /
>         ( address-literal [ string ] )

Yes, that's my reading as well.

> But there is no need for any of that qualifying nonsense with a 
> domain name.  One might imagine passwords, etc., out there, but 
> the design model says that those are expected to go in a 
> separate command, on MAIL FROM, etc., after the server 
> authorizes (or requires) sending them.
> 
> Is that what is wanted, or should the ABNF be
> 
>        Domain = ( (sub-domain 1*("." sub-domain)) /
>           address-literal ) [ string ]

Remember that the syntax has to work for plain old HELO as well,
which was never intended to have additional parameters.  Also, as
pointed out, even with EHLO, you can't send anything additional
until after you've seen a 220 that says it's OK.  As such, the only
case where 2821 suggests a [string] is needed is for hints for an
address literal, so I'd prefer the first variant you listed.

--==_Exmh_-1465556040P
Content-Type: application/pgp-signature

--==_Exmh_-1465556040P--

Date: Fri, 02 Jan 2004 17:41:26 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], "B. Johannessen" <[email-masked]>, [email-masked]
Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
Sender: [email-masked]

Arggh.

First, the ABNF is wrong given the text and, on behalf of myself 
and the several people who checked it, my/our apologies.

That said, there seems to be an open question on which 
discussion/ advice would be useful.

The text, as written, implies that the ABNF should have been

       Domain = (sub-domain 1*("." sub-domain)) /
         ( address-literal [ string ] )

and, while my memory is dim, I think that is exactly what was 
intended.  In other words, if you say

       EHLO [1.2.3.4]
then you are strongly encouraged to say

       EHLO [1.2.3.4] unnamed host, for questions mail to 
[email-masked]

But there is no need for any of that qualifying nonsense with a 
domain name.  One might imagine passwords, etc., out there, but 
the design model says that those are expected to go in a 
separate command, on MAIL FROM, etc., after the server 
authorizes (or requires) sending them.

Is that what is wanted, or should the ABNF be

       Domain = ( (sub-domain 1*("." sub-domain)) /
          address-literal ) [ string ]

??

--On Friday, 02 January, 2004 21:33 +0000 "B. Johannessen" 
<[email-masked]> wrote:

> I'd say it's section 4.1.1.1 that's b0rked. If you allow free
> form "information" after the FQDN/address-literal it makes it
> impossible to parse (ESMTP) parameters for EHLO.

Bob, I don't see the issue you are trying to get at.  The model 
is that the server/ SMTP receiver announces itself with a 
largely information-free sanity-check banner, e.g.,
    220  bogus.domain.name here
the client / SMTP sender opens the actual mail processing by 
asking for capabilities and indicating that it can accept them 
with, e.g.,
    EHLO mail-client.example.com
then the server responds with capabilities and, after that, the 
client starts using/ requesting whatever subset of the 
capability list makes sense.  There has never been any model for 
the client to announce anything to the server at EHLO time, nor 
to put any ESMTP parameters there.

    john

--On Friday, 02 January, 2004 13:50 -0500 
[email-masked] wrote:

> Kari Hurtta pointed this out to me on the Sendmail development
> list while we discussed what the Right Thing To Do for
> hostname-challenged machines was..
>
> Section 4.1.1.1 says:
>
>    These commands are used to identify the SMTP client to the
> SMTP     server.  The argument field contains the
> fully-qualified domain name    of the SMTP client if one is
> available.  In situations in which the    SMTP client system
> does not have a meaningful domain name (e.g., when    its
> address is dynamically allocated and no reverse mapping record
> is    available), the client SHOULD send an address literal
> (see section    4.1.3), optionally followed by information
> that will help to identify    the client system.  y The SMTP
> server identifies itself to the SMTP    client in the
> connection greeting reply and in the response to this
> command.
>
> It then goes on:
>
>    Syntax:
>
>       ehlo            = "EHLO" SP Domain CRLF
>       helo            = "HELO" SP Domain CRLF
>
> And in 4.1.2 we find this ABNF:
>
>       Domain = (sub-domain 1*("." sub-domain)) /
> address-literal       sub-domain = Let-dig [Ldh-str]
>
>       address-literal = "[" IPv4-address-literal /
>                             IPv6-address-literal /
>                             General-address-literal "]"
>             ; See section 4.1.3
>
> By my reading, this allows 'ehlo F.Q.D.N' and 'ehlo
> [add.ress.lit.eral]', but there is no way to actually generate
> 'optionally followed by information' using the ABNF as written.
>
> Am I caffeine-challenged, or is the ABNF actually borked on
> this?
>

Subject: Re: RFC2821, section 4.1.1.1 and HELO/EHLO
From: "B. Johannessen" <[email-masked]>
To: [email-masked]
Organization: http://db.org/
Date: Fri, 02 Jan 2004 21:33:40 +0000
Sender: [email-masked]

On Fri, 2004-01-02 at 18:50, [email-masked] wrote:
> By my reading, this allows 'ehlo F.Q.D.N' and 'ehlo [add.ress.lit.eral]', but
> there is no way to actually generate 'optionally followed by information' using
> the ABNF as written.
> 
> Am I caffeine-challenged, or is the ABNF actually borked on this?

I'd say it's section 4.1.1.1 that's b0rked. If you allow free
form "information" after the FQDN/address-literal it makes it
impossible to parse (ESMTP) parameters for EHLO.

	Bob

-- 
-=[ B. Johannessen | [email-masked] -=- http://db.org/ | +4797152009 ]=-
-=[ Mail & Spam - News, Drafts & Standards - http://db.org/blog/ ]=-

To: [email-masked]
Subject: RFC2821, section 4.1.1.1 and HELO/EHLO
From: [email-masked]
Date: Fri, 02 Jan 2004 13:50:06 -0500
Sender: [email-masked]

--==_Exmh_2131767092P
Content-Type: text/plain; charset=us-ascii

Kari Hurtta pointed this out to me on the Sendmail development list while we
discussed what the Right Thing To Do for hostname-challenged machines was..

Section 4.1.1.1 says:

   These commands are used to identify the SMTP client to the SMTP 
   server.  The argument field contains the fully-qualified domain name
   of the SMTP client if one is available.  In situations in which the
   SMTP client system does not have a meaningful domain name (e.g., when
   its address is dynamically allocated and no reverse mapping record is
   available), the client SHOULD send an address literal (see section
   4.1.3), optionally followed by information that will help to identify
   the client system.  y The SMTP server identifies itself to the SMTP
   client in the connection greeting reply and in the response to this
   command.

It then goes on:

   Syntax:

      ehlo            = "EHLO" SP Domain CRLF
      helo            = "HELO" SP Domain CRLF

And in 4.1.2 we find this ABNF:

      Domain = (sub-domain 1*("." sub-domain)) / address-literal
      sub-domain = Let-dig [Ldh-str]
   
      address-literal = "[" IPv4-address-literal /
                            IPv6-address-literal /
                            General-address-literal "]"
            ; See section 4.1.3

By my reading, this allows 'ehlo F.Q.D.N' and 'ehlo [add.ress.lit.eral]', but
there is no way to actually generate 'optionally followed by information' using
the ABNF as written.

Am I caffeine-challenged, or is the ABNF actually borked on this?

--==_Exmh_2131767092P
Content-Type: application/pgp-signature

--==_Exmh_2131767092P--