Recommended method of spam quaranteening

This basically involves using friends silent mode instead of the spam held mode as the primary method to quarantine email identified as likely spam.

The advantages of this are:

• Reduced false positives by making use of the the Friends whitelist.
• Spam folder management using clickable links in the status email allowing management from your email client.
• In status email, can show you just the new messages received since the last status email was last sent.
• Provides a single quaranteen location for the storage of all messages identified as possible spam.
• This spam quarantine location can be made available through imap and may be renamed if needed.

Why change?

The primary reason is to make use of the Friends whitelist. Traditionally surgemail has had two ways of storing spam before it gets to your inbox.
1. "Friend pending" folder where messages were held pending "friends bounce" confirmation that the sender was human.
2. "Spam Held" folder that all messages with a high spam rating were placed.

The most effective way to configure this has been to use a friends bounce if over a certain spam rating. This way all spam will be stored in the "(Friends) Pending" folder and if senders of messages that get placed in the "Pending" folder reply to the bounce the email will be immediately delivered to your inbox.

Some people try and avoid sending out of challenge / response bounces out of personal preference. In this case you would likely have the friends system disabled, and have messages end up in the "(Spam) Held" folder based on spam rating. The friends system intrinsically allows email based on a whitelist of valid senders. If the friends system is in the disabled mode, mail from people you have been mailing from a long time would often end up in the "Held" folder - just because in this case the message happened to look like spam.

In the case where you are trying to avoid sending challenge respose emails it works much better if you use the friends mode "silent". This way all email over a certain spam rating is placed in your "(Friends) Pending" folder (displayed in surgeweb as the "Spam" folder). This way email from people you have mailed in the past will always match the friends whitelist and never get identifed as likely spam.

The second core reason is that you can use the new html formatted status emails to manage your quarantined messages directly from within your normal email client.

New format html spam status email

The old plain text status email reporting on the status of the friends pending (and spam held) folders has been replaced by an html formatted email with more information on the email messages that have been detained in the quarantined in the "Spam" [aka "(Friends) Pending"] folder.

This email provides clickable links so spam email messages can be managed directly from your favourite email client without needing to manually log into user.cgi / webmail / surgeweb to process messages identified as spam. This is particularly useful if you are using surgemail in a surgewall configuration for spam filtering. You will need to enable the status email on the log page in user.cgi and you need to be running a recent version of surgemail (4.0v-8+).

The new formatted html email by default allows you to take the following actions on messages using a single click:

• Deliver messages to your inbox, training the from address in your friends list to ensure future messages from this user are always delivered without being marked as spam
• View the message safely without delivery to your inbox
• Delete the message
• Permanently block all email from this sender
• There are also options to enable spam reporting and Spam folder purging from this page

Another feature of the html status email is that it displays all the relevant addressing headers if they are different from the From header to help identify mail with faked headers. In particular "Reply-To" and "Return-Path".

This status email may be customised and the basis of this email is the surgemail/status_html.eml file. The status emails can also be sent at a particular time of the day using the g_spam_status_hour setting.

Surgemail will start globally using the html status email by default. Individual accounts can be configured to send the old style plain text or new format html status email using the user.cgi log page. In addition the global default to start usign the html status email can be disabled using the setting:

  g_friends_old_status_email "true"

Updating to new Default spam handling settings

To configure the new global default behaviour all you need to configure is g_friends_default_mode. This can be configured several different ways:

  g_friends_default_mode "silent"

  g_friends_default_mode "list"

  g_friends_default_mode "smite"

This will enable this friends for newly created accounts. For new surgemail installations g_friends_default_mode silent is now the default friends mode.

In addition to this, you will need to make sure you have no global defaults or domain defaults defining spam hold settings or disabling Friends whitelist. You will be warned during the tellmail conversion below if you have any of these.

Plus optionally if you are using surgeweb convert the relevant surgeweb settings.

Lastly there is one further setting (version 4.3b-2+) you may want to use to consolidate the changes:

  g_spam_hold_hide "true"

• Hides the Spam hold and vanish settings for end users (Spam reject is still displayed), and displays a warning in the spam settings to use the Friends based quarantining settings instead. Spam hold and vanish settings are still displayed for server admin.
• In filtering and exceptions renames Request to Quarantine and treats spam hold exactly the same as the Quarantine setting (ie the option that was previously named Request). Even though it was previously named request it only sent a challenge email if one of the friends modes was configured that sends challenge emails, so quarantine is more appropriate.
• Some rewording in the Friends settings

Changing spam settings for existing users

Changing the defaults above does not affect accounts where changes have been made manually by users at the user.cgi level.

A tellmail command has been implemented (surgemail version 4.2g-33+) to aid the conversion process by allowing the admin to convert all users at once. The syntax is:

  tellmail held2pend (global|mydomain.com|email@mydomain) [apply] [nocheck]

Where:

• apply - required to actually make changes. Whithout this it simply tells you what it would convert.
• nocheck - disables the checks for spam_held being enabled by default or friends being disabled by default. Using this flag makes testing the help2pend functionality on a single account easier, but normally before use on multiple accounts these checks should get passed rather than ignored using 'nocheck'.

>~ tellmail held2pend global

Converting accounts using spam_held to using friends_silent.
WARNING: Running test only, rerun with APPLY parameter to actually convert accounts.

Processing domain (mydomain.com):
user1@mydomain.com: 283 msg; friend mode=disabled->silent score=5->10 add_auto(enabled); {wrap}
                             spam H/R/V= 10/0/0 -> 0/0/0
user2@mydomain.com: 15 msg; friend mode=list score=5->8; spam H/R/V= 8/0/0 -> 0/0/0
user3@mydomain.com: 94 msg; friend mode=smite score=7; spam H/R/V= 15/0/0 -> 0/0/0

Processing complete.
>~

• User1 had friends disabled and was using spamheld on a scoring of 10 or higher. This has been converted to using friends silent mode, at rating of 10+, with addresses of outbound mail getting added to the whitelist.
• User2 was already using friends whitelisting (not using the score value at all though), but now the spam gets stored in the friends folder rather than the held folder at a rating of 8+.
• User3 was already using friends challenges at a rating of 7-15, with 15+ getting stored in spam held. Now any mail with a rating of 7+ gets sent a challenge and is stored in the friends pending folder.

>~ tellmail held2pend global apply

What the conversion actually does is:

1. Switch friends disabled or list modes to list or silent dependent on the spam_held rating. Other user configured friends modes are retained.
2. Use the lowest of the relevant processing ratings (held/vanish/bounce or friends smite rating) and us this as the friends smite rating. POssibly keeping the spam_reject setting if relevant.
3. Enable friends whitelist addition based on outbound authenticated smtp.

Two additional relevant tellmail commands:

held2pend_email Set spam held email frequency for all accounts. days="days between emails" with special values 0="disabled" and -1="server default"

tellmail held2pend_email (global|mydomain.com|email@mydomain) days=n [apply]

pending_release (global|mydomain.com|email@mydomain) (nofriend|all) [cutoff_rating] [apply]

Word of warning

Surgemail will automatically switch to using the html status email in versions (4.0v+), and will only send this if there are messages to report on. This was recently identified as having the nasty and unintended side effect in surgemail versions (approx 4.0v to 4.2g-26) that for accounts using the status email to report on messages only in the spam held folder (and not using friends) the status email will appear to stop getting sent.

This is resolved in version 4.2g-27+. This version also introduces per account control over whether the new format html or old format plaintext status email gets sent.