Central email filtering

Despite the best efforts of the Computing Service,
vast quantities of junk email (viruses and spam)
continues to arrive every day and often outnumbers genuine wanted email.
Unfortunately, it is difficult to block all such messages without
there being a real risk of losing genuine wanted email;
often only the recipient can determine whether a message is wanted or not.

General advice about dealing with junk email,
including harassing email,
has been published in previous Newsletters and can be found at
http://www.cam.ac.uk/cs/email/

Meanwhile, the Computing Service is working on improving the filtering
systems we currently have,
for dealing both with junk email and with the specific problem of
virus-bearing email.


Background

The Computing Service's central email systems are roughly divided
into two sets of machines, Hermes and PPSW.

Hermes is the central email store; it also handles outgoing email
for users via smtp.hermes.cam.ac.uk.

PPSW is the central email relay that deals with
@cam forwarding,
@lists mailing list distribution,
managed mail domains,
and "hubbed" forwarding to and from departmental and college email systems.

Some email systems in the University,
including Hermes and the Central Unix Service,
do not use the PPSW hub but send and receive email directly to and from
the rest of the Internet.


The status quo

The differing functions of Hermes and PPSW outlined above
have resulted in different policies on each system (Hermes being stricter),
as follows.

Both Hermes and PPSW reject email from machines on the Internet
that are open to abuse by spammers
according to a list known as the JANET/MAPS RBL+
(see http://www.ja.net/CERT/JANET-CERT/mail/mail-abuse/mail-abuse.html);
we also maintain our own blacklist of spam email sender addresses
(although because of the prevalence of forgery this is not very effective).

In addition to that, Hermes (but not PPSW) has a rudimentary virus
filter designed to block email worms (such as Klez) that exploit
weaknesses in Microsoft Outlook etc.

Finally, Hermes users can add their own filters using the menu system or
via webmail (see http://www.cam.ac.uk/cs/docs/faq/e6.html),
and advanced users can write an
Exim filter (see http://www.exim.org/exim-html-4.10/doc/html/filter_toc.html)
to handle email according to more complicated rules.


Work in progress

The increase in junk email means that stricter filtering is required everywhere,
not just on Hermes.
We are currently working on an improved filtering system for PPSW
in addition to the existing facilities.

It will use a commercial virus scanner
(see http://www-tus.csx.cam.ac.uk/virus/) which,
unlike the existing filter on Hermes,
will be much better at keeping up with the arms race
and will also be able to deal with viruses inside attachments.

It will also use SpamAssassin
(see http://spamassassin.org/)
to give each email a score based on how it matches a set of spam signatures.

The precise details of the filtering policy have not been pinned down yet,
and will change based on practical experience,
but the general rule is that no real email will be discarded --
it will either be delivered to its recipient (possibly after disinfection)
or returned to its sender.

The spam score will be just an annotation on the email and will not
cause PPSW to reject it.
Because it is not perfectly accurate and because people disagree on
the acceptable level of mis-identified email,
filtering based on the spam score will be left for end-users to
configure as it suits them.
Options will be added to the Hermes menu system and webmail interface
to make such end-user filtering easy to set up.

Since the main filter will run on PPSW it will affect all email in the
University except for departmental or college systems that do not hub
through PPSW.
When the filter is implemented Computer Officers may want to
reconfigure such systems to hub through PPSW so that they may benefit
from it.
We will be doing this with Hermes
(including smtp.hermes.cam.ac.uk)
which will allow us to get rid of the old virus scanner
and simplify away all the differences in policy.
The Central Unix Service is likely to follow suit.
On the other hand it will be possible for departments and colleges to
opt out of the filter even if they hub through PPSW.


$Cambridge: hermes/doc/newsletter/2003-01-filtering.txt,v 1.13 2003/01/15 15:54:58 fanf2 Exp $