Spammail understand header and move mails

Various scanners can be used to scan incoming and outgoing mail. For example, ISPConfig uses SpamAssassin, ClamAV and Amavis. The individual threshold values (more on this later) can be defined individually.

To prevent spammers from appearing in the inbox, you can automatically move mails to a separate folder based on an individual spam value. This makes spam mails visible via an IMAP client or a webmailer and reduces the risk of not seeing a mail mistakenly identified as spam.

Caution: when the mails are picked up using POP3, the moved mails will not be retrieved. They are no longer in the inbox, but in a subfolder. This moving is always done directly on the server.

Scan Procedure

  • postfix receives the mail
  • and relays the mail to amavis
  • on the basis of different spam rules (see also current rules for SpamAssassin from schaal @it) the mail gets different information in the header
  • Amavis sends the mail back to Postfix

Structure of a mailheader

We deliberately leave out most of the entries from a mailheader because they are not relevant for detecting or moving spam. The essential lines of a mail header see e.g. like this:

 X-Virus-Scanned: Debian amavisd-new at scan.schaal-it.net
 X-Spam-Flag: YES
 X-Spam-Score: 7.039
 X-Spam-Level: *******
 X-Spam-Status: Yes, score=7.039 tagged_above=3 required=5 tests=[BAYES_50=0.8, DATE_IN_PAST_24_48=1.34, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RP_MATCHES_RCVD=-0.001, SCHAALIT_HEADER_964=5] autolearn=no

X-Virus-Scanned

The first line shows which scanner was used. If you want other information, you can adjust the Amavis-Config accordingly. For Debian or Ubuntu, this would be $ X_HEADER_LINE in /etc/amavis/conf.d/20-debian_defaults.

X-Spam-Flag

If this is set to YES, the mail was classified as spam.

X-Spam-Score

The X-Spam score is the value that SpamAssassin has determined based on its filter rules for the mail. In ISPConfig, you can define the value from which a mail should be marked as spam (5.00 for Trigger happy) under Policy – Spamfilter policy – Tag Level in the “SPAM tag2 level” field.

X-Spam-Level

This is the integer value of X-Spam-Score in * – in our case, the score is 7.039 and thus there are seven asterisks. You can use the value with your own filter, if you do not want to take the X-Spam-Score.

X-Spam-Status

Yes, score=7.039 this is the same as in X-Spam’s flag and X-Spam score.

tagged_above=3 specifies the value from which these lines listed here should be inserted into the header. For ISPConfig, this is the “SPAM tag level” for the Spamfilter policy.

required=5 is the value from which the mail is then marked as spam and the X-Spam flag is set to YES. For the Spamfilter policy, this is the “SPAM tag2 level”

tests= shows you which individual rules SpamAssassin has applied and how many points have been set for this mail:

BAYES_50, DATE_IN_PAST_24_48 and SCHAALIT_HEADER_964 have increased the value, the signing with DKIM has reduced the score by a total of 0.1 (DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU) and RP_MATCHES_RCVD has decreased the value by 0.001.

overall, therefore:

0.8 + 1.34 + 5 = 7.14

0.1 – 0.1 – 0.1 = -0,1

– 0.001

And this brings us to score of 7.039

If you are interested in the contents of the tests, you can find the rules in different folders in /var/lib/spamassassin. As an example we take the rule SCHAALIT_HEADER_964:

header SCHAALIT_HEADER_964 From =~ /\@pkv.*\.(online|net|com|org|co|info|de)/
describe SCHAALIT_HEADER_964 schaal @it Spam
Header-964 score SCHAALIT_HEADER_964 5

The rule applies to the from line in the mail. In doing so, a RegEx is used, which spans various spam mails on the subject of german private health insurance.

Move the mail to the spam folder

After Postfix has returned the mail from Amavis (see Scan Procedure), the mail is either sent, or – if the recipient is one of our users – delivered via Dovecot into the mailbox.

The local delivery of the mail by Dovecot is one of the decisive points: if the mail was previously marked as spam by SpamAssassin, it is not moved to the inbox, but to the subfolder Junk. For ISPConfig, the checkbox for the email account in the email filters must be set to “Move Spam Emails to Junk directory”.

Basics about the filters with ISPConfig

  1. A spam filter must be selected for the domain or mailbox. If a filter is defined for the domain and the mailbox, the settings from the mailbox are in place. If only the domain has a filter, it’s also used for the mailbox. In short: Mailbox overwrites domain – this is the only way, to define individual policies for each mailbox. If you use an alias, the policy for the domain applies.
  2. The selected spam filter must be defined in such a way that the corresponding values also “fit” to the mail. If our filter had a “SPAM tag2 level” of 10, the mail would not be marked as spam.
  3. Spam-mails will only be moved if “Move Spam Emails to Junk directory” is set.
  4. Only the administrator can change or create policies. If every user had the possibility, this would affect all the mailboxes that use this filter. If you are in doubt, you better suggest an additional guideline.

Current rules for SpamAssassin from schaal @it

Current rules for SpamAssassin from schaal @itWe regularly publish new rules for the spamfilter SpamAssassin of the Apache Software Foundation Apache Software Foundation. Of course, you can also use the rules for the ISPConfig spam filter settings.

Most of our rules have a score of 5.

Our rules can be used free of charge. For this, only our channel has to be added to the SpamAssassin-Config or our script has to be installed. Please do not forget to restart SpamAssassin after installing or updating rules.

SpamAssassin uses DNS checks to detect new rules so that the script can be called every hour or at least dayly without generating a significant load.

Install our ruleset:

sa-update --nogpg --channel sa.schaal-it.net

To keep the rules up-to-date, just install a simple shell-script in /etc/cron.hourly:

cd /etc/cron.hourly && wget sa.schaal-it.net/sa-update && chown root.root sa-update && chmod 755 sa-update

This will install this script:

#!/bin/sh
# schaal @it
#
# Simple script to update SpamAssassin

SYSLOG_TAG=sa-update

compile=0

logger -d -t $SYSLOG_TAG "Start SA-Update"

sa-update --nogpg
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi


sa-update --nogpg --channel updates.spamassassin.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sa.schaal-it.net
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

if [ $compile -eq 1 ]; then
	logger -d -t $SYSLOG_TAG "SA-Update found"
	sa-compile
	/etc/init.d/amavis restart
else
	logger -d -t $SYSLOG_TAG "No SA-Update found"
fi

To add more rules to the spam filter simply add one or more blocks before if [ $compile -eq 1 ]; then:

sa-update --nogpg --channel updates.spamassassin.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sought.rules.yerp.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel spamassassin.heinlein-support.de
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

ISPConfig Backup-Space remote

You can use our BackupSpace to run ISPConig with this as an external backup space.

ISPConfig offers you the possibility to save the backups directly to an external storage. It makes no sense to back up relevant data for recovery directly on the server, which should be restored in case of any problems.

To do this enable the option  “Backup directory is a mount” and create a small shell-script.

SSH-Key

If there is no SSH key on your server (/root/.ssh/id_rsa.pub), you must first create a key pair with ssh-keygen:

ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

Then create a shell user for your backup space and enter the key from /root/.ssh/id_rsa.pub there.

Mount-Script

The script /usr/local/ispconfig/server/scripts/backup_dir_mount.sh is used to mount the external memory before backups. With our backup space you only need two lines:

#!/bin/bash
sshfs YOUR-SSH-USER@YOUR-ID.backup.schaal-it.com:/backups /var/backup

Finally, set the right permissions for the script:

chown root.root /usr/local/ispconfig/server/scripts/backup_dir_mount.sh
chmod 700 /usr/local/ispconfig/server/scripts/backup_dir_mount.sh

If you want, you can unmount the space after the backups. Just create the script /usr/local/ispconfig/server/scripts/backup_dir_umount.sh:

#!/bin/bash
umount /var/backup

Whether you’re using umount or not, it’s up to you. If the backup space is not mounted, ISPConfig will always attempt to mount the storage. This is not only the case with backups, but also when restoring individual backups.