Re[2]: [Asrg] REDIRECt to ASRG RE: Reputation systems

[Pete McNeil <madscientist at microneil.com>]

I'm copying Markus on this:
Markus, can you provide a legend for the tests on the SpamTest Quality
Analysis page? It is unclear what some of the test names represent.

More below...

On Tuesday, August 17, 2004, 12:51:59 PM, Ryan wrote:

RM> [Pete McNeil]
>> Here is a link to the analysis:
>> 
>> http://www2.spamchk.com/public.html
>> 
>> Hope this helps,
>> _M

RM> This is very useful, and most of what I was looking for, but I'd still
RM> like to see an analysis done with hand-filtered email.

That might be useful, but I suspect the differences in the resulting
data would be "in the noise". The system presented in the analysis is
tuned also by a manual review of the content being filtered - so the
final result used to drive the analysis very closely approximates a
manual accounting on an ongoing basis.

There are bound to be some errors, but from what I've seen they are
vanishingly small.

<snip/>

RM> The numbers seem to indicate that IP-based blacklisting is essentially
RM> useless without further content filtering. None even came close to
RM> filtering 75% of spam, and some caused hundreds or thousands of false
RM> positives.

This is true of many single tests, but that's not the way they are
used.

Declude can achieve very strong numbers by combining a broad
list of DNSBLs and a few simple content tests (mostly header and
envelope analysis) in a weighted framework.

[ Here is a collection of reports that used to be produced regularly by
Declude (At the bottom of the page beginning with March):

http://www.sortmonster.com/MessageSniffer/Reviews/Reveiws.html ]

Declude's analysis has consistently shown that by combining the
available DNSBLs in a weighted framework it is possible to achieve
very good results.

That said, content analysis is _very_ strong. The best solution is and
will probably continue to be a diverse combination of mechanisms.

RM> I wonder if the same poor performance will be the output of
RM> domain-name-based reputation systems after MARID's work is widely
RM> deployed.

Domain-name based systems will drive spammers to rapid migration and other
obfuscation techniques. SURBL seems to be prompting this already. I
think that domain-name based systems will probably show a higher
performance than IP based systems, but they will still fall far short
of content analysis.

There are a number of other attacks on the rise also - for example, it
is a problem with SURBL that candidate URI extracted from the message
must be tested individually. As a result, spammers are inserting large
numbers of links and patterns that will be extracted as candidates so
that SURBL type systems can be overloaded. Rather than checking a few
URI against the SURBL DNS a comprehensive test needs to test dozens or
even hundreds... Since that's unreasonable, many SURBL implementors
check a subset of candidates at random which solves the resource
problem at the expense of accuracy. (then of course there is poisoning
as well...)

(BTW: our scanner can address all possible URI simultaneously - we've
been using URI, fragments, and patterns from very early on with great
success so clearly the concept is powerful. Implementation is the
challenge.)

Content analysis systems like Message Sniffer can filter abstract
patterns and variations of spammy domains so the these heuristics can
be more resilient - often predicting URI before they occur in the
wild. Other systems which concentrate on domain names will have the
problem that the must always (bad word) see the domain in the wild
before they can establish a reputation for it.

Content analysis mechanisms _can be_ slow and resource intensive,.. for
example - large rule sets built from regular expressions can require a
lot of computing power, and a lot of manpower for maintenance. Then
again, that's not the only way to do it. Our pattern matching engine,
for example, can apply many 80,000 unanchored patterns to a typical
email in well under 100ms on outdated hardware (p2/450).

--- one of the key benefits of reputation systems will be to provide
more live data on the "white side" of the filtering problem - to help
prevent false positives thereby allowing "black side" filtering tools
to be more aggressive. Combining a wide variety of tests is crucial.

(BTW: Message Sniffer, while implemented as a content scanner, does
apply a range of testing methods through that mechanism - so it's not
entirely what you would expect from content analysis and doesn't
really represent the performance of a strict content analysis regime.
For example 10-25% of an active rulebase can contain IP rules that
match header content. This is not unlike a DNSBL except that it is
implemented as a content scan rather than a DNS lookup.)

RM> Do you have a key that explains what each of the test abbreviations are
RM> on this page? Some are obvious, others are not.

I'm sure Markus can address this - I mentioned it above.

_M




_______________________________________________
Asrg mailing list
Asrg at ietf.org
https://www1.ietf.org/mailman/listinfo/asrg