KMail and bogofilter HowTo

Thomas Strauß, thst<at>strauss-it<dot>de

Have a spam-free Incoming and get back to the important stuff :-)

Last Changed: 11.08.2003

	- there has been a bad typo in the commandlines to teach your spam database
      to mark messages as "no spam" you must use the "-n" switch. Currently both
      commandline switches basically do the same thing: Mark the mails as spam.
      Sorry about that fatal typo and thanks to Hendrik Naumann ( hn75<at>gmx<dot>de )
      for finding it.

11.07.2003:
	- added Idea from brian.harnish<at>verizon<dot>net to avoid procmail and use less
      KMail filters if you implement your own filtering, you should use his idea,
      if you dont care about procmail.
	- added information about teaching bogofilter
    - this text gets too long to keep it unstructured.... next time with chapters :-)


Procmail-Solution:
==================

Required Software:
- procmail
- formail
- bogofilter
- KMail

The process is as following:

1. first of all, mark incoming messages as "unchecked". This is not
   explixitly required, but helps debugging and avoids doublechecking.
2. then pipe all unchecked messages through procmail/bogfilter/formail.
   The pipeline is controlled by procmail.
   formail adds a header "X-SPAM: yes" if it is spam
3. forward all messages marked as "X-SPAM: yes" to the spam-database

This is accomplished with three kmail filters:

1. BogoFilter Mark:
	<Message> contains not "X-SPAM-TESTED"
    <Add Header> "X-SPAM-TESTED" value "no"
    keep on processing

2. BogoFilter:
	<X-SPAM-TESTED> contains "no"
    pipe through <PATH TO PROCMAIL_EXE>
	<remove Header> "X-SPAM-TESTED"
    <Add Header> "X-SPAM-TESTED" value "yes"
    keep on processing

3. BogoFilter remove Spam
	<X-SPAM> contains "yes"
    <move to folder> "spam this"
    end processing

The required ~/.procmailrc script looks like this:

:0HBf:$LOCKFILE
* ? /opt/software/bogofilter/bin/bogofilter
| formail -a "X-SPAM: yes"

where LOCKFILE contains the name of a lockfile and is optional
and you should substitute /opt/software/bogofilter/bin/bogofilter by the path
to bogofilter in your system you could easily enhance the script to mark "non spam"
with "X-SPAM: no" as such by adding

:0Ef:$LOCKFILE
| formail -a "X-SPAM: no"

One remark: You should define your own X-Headers. Dont use the ones I use, as this
would give a spammer the potential possibility to mark its spam as "spam-checked"
and by this wiggle through your filterchain unseen.


Non-Procmail-Solution
=====================

Another possibility to control bogofilter with KMail has been pointed out by
bharnish<at>technologist<dot>com, I quote him because of my lazyness :-)
He uses the fact, that I have in fact overseen, that bogofilter can add a
headerline, marking it as "SPAM" or not, thus removing the need for the construct
above, which basically only adds the headerline to filter it out afterwards.


Brian wrote: ----------------------------------------------------------------
[...] you suggested not using the same names for the rules. I have an
implementation that doesn't require different names, and is
a step shorter (not that mine is better, there could be some flaw I didn't
notice):

Rule 1:
<X-Bogosity> doesn't contain "Yes, test=bogofilter"
pipe through "bogofilter -epu"
keep processing

Rule 2:
<X-Bogosity> contains "Yes, test=bogofilter"
move to folder "Caught Spam"
Mark as read
end processing

The only way it won't go through bogofilter is if the spammer sets the header
for me saying it is spam. In that case, it goes into the spam folder like
everything else.

[...]

------------------------------------------------------------------------------

Teaching Spam from No-Spam messages
===================================

After bogofilter sorted all the spam out, you should add the spam and non-spam
messages to the spam-database. I do this using a cron entry to run these two commands:

Teach spam messages:
( /opt/software/bogofilter/bin/bogofilter -vv -s <~/Mail/spam\ this ) && ( echo >~/Mail/spam\ this )

to teach no-spam messages:
( /opt/software/bogofilter/bin/bogofilter -vv -n <~/Mail/spam\ this\ not ) && ( echo >~/Mail/spam\ this\ not )

kmail is annoyed by removing the contents of the mailbox file, but will recreate the index
nicely and even offer you to not bother anymore.
Be warned: I suspect it to eat incoming mails, while the requester is open.
Therefore I strongly suggest, you remove the warning-feature.

If your spamdatabase reflects your incoming mail (no wrong classified mails anymore),
you can add the "-u" flag to bogofilter, This will add the mail to the (no)spam-database immediately.
The required ~/.procmailrc script looks like this:

:0HBf:$LOCKFILE
* ? /opt/software/bogofilter/bin/bogofilter -u
| formail -a "X-SPAM: yes"

Brian's solution already uses the -u flag. Just leave it out, if your database is not sufficiently filled.