Spam and Blocklists

Note If you came here to learn how to get your IP delisted from Symantec Cloud's email filter: Sorry, you can't.

Email Filtering

Spam is annoying because it distracts and makes it harder to identify email messages that need attention. Unwanted emails also frequently have attachments with content that can damage your computer, or they can be written in a way that encourages the recipient to part with money or disclose personal information. So you generally want to have these filtered out before they reach your inbox, or at least have them placed in a spam folder.

Anyone who runs an email server knows that dealing with spam is not a trivial problem. Filtering email so that only non-spam and non-attack email reaches the user's inbox is requires multiple layers of defenses, and there is always a balance to be maintained in order to avoid legitimate email being stopped by spam filters.

Blocklists

Email blocklists are an important part of spam filtering systems. Blocklists are basically lists of IP addresses of known spammers. When an email arrives at a mail server, the server can compare the sender's IP address against one or several blocklists; if the IP address is found on a blocklist, the receiving server can then prevent delivery of the email or use the blocklist information as input to a spam scoring system. Used intelligently this can be a very effective way of reducing spam.

There are many publicly available blocklists that can be used by anyone who runs an email server, e.g. Spamhaus, SpamCop, Abuseat CBL, and Barracuda RBL. Proprietary blocklists are also an integral part of commercial spam and email security solutions.

Listing and delisting

Compiling a list of IP adresses of bad email senders seems trivial, but keeping a blocklist current is actually a demanding job. There is a huge number of spammers out there, and the IP addresses they use change constantly so the blocklist must be continuously updated.

Adding an IP to a blocklist requires some way of identifying spam, but this in itself is difficult since the distinction between spam and nonspam is to a large extent a matter of personal preference. For example, a bulk email message containing product offers may be actively wanted by some recipients, while others regard it as spam. Typically this depends on whether the recipient has actively requested to be added to a distribution list or not, but the email server has no access to information about opt-ins and so cannot use it to determine if an email is spam.

Because there is no hard definition of "spam" it is also relatively easy to end up on a blocklist by mistake. A classic example is an email user who subscribes to a mailing list, forgets about it, and then hits the "spam" button instead of unsubscribing from the list. It is also easy to become listed if dumb autoresponders send out-of-office replies in response to spam messages with forged "from" addresses.

An IP address may also send spam for a while and then stop. An email user may have been abusing the service, or a hacked WordPress website on a shared server has been sending spam, causing a shared IP to be listed on one or several blocklists. You may also find yourself on a blocklist if you are renting a server or a VPS and happen to get an IP address that was previously used by a spammer. Managing a blocklist is a continous process of adding and removing IPs from the list, and the usefulness of a list depends entirely on how well this process is managed.

Delisting process

Once the IP address of an email server is added to a blocklist, the server will have difficulty delivering email to other servers that use that blocklist as part of their spam filtering process. Email senders receive non-delivery responses and the server admin will see delivery error messages in the server logs.

If the server that is rejecting the email has been configured to use the blocklist correctly, the rejection response will contain a reason for the rejection along with a URL to a web page with further information. That web page will then contain additional information about the reason for the block, and instructions on how to request removal of an IP address from the list (see an example here).

It is important to address the reason that the IP address was listed before requesting removal. If your server is in fact sending spam and continues to do so, your request may be denied, or the IP address will quickly be listed again. Repeated removal requests may be considered abuse by the blocklist admins and may lead to future delisting requests not being considered.

There are various tools that lets you check an IP address against multiple blocklists, such as MX Toolbox. There are also services that can assist with the delisting process, for a fee.

Large freemail providers such as Gmail, Hotmail, and Yahoo maintain their own blocklists, and the process for getting delisted is not always clearly stated. See this blog post for an example of how to get delisted if Hotmail is blocking your IP.

Anyone can create and maintain a blocklist and allow others to use it. Occasionally you may find that your IP has been added to some obscure blocklist, and that the blocklist owner wants you to pay for removal. My advice would be to not pay such fees and ignore the blocklist, but use your own judgement.

Symantec Cloud Blocklist

After recently moving our own email server to another provider and getting a new IP address, we started seeing delivery errors similar to the following in our delivery logs:

SMTP error from remote mail server after initial connection: 501 Connection rejected by policy [7.7] 3108, please visit www.messagelabs.com/support for more details about this error message.

This is a typical example of the sort of error message that you see if your IP address is on a blocklist. However, the URL in the reject message is invalid, it simply redirects to Symantec's web page with a "Page not found" message. This obviously complicates the delisting process.

The rejected emails were sent to recipients in large companies, so we decided to pursue the matter further. This did not turn out to be easy - Symantec has a lot of different products and services - but we eventually found out that Messagelabs must have been acquired by Symantec at some point and their service integrated into one of their "cloud" services. So we sent a polite request to support.cloud@symantec.com requesting their assistance in the matter.

About 24 hours later we got an equally polite and utterly unhelpful response from Symantec:

Our best advice at this point would be to refer you contact any domains that are client of ours that you are sending to so that they may investigate and then liaise with Symantec.cloud where appropriate.

So in order to get our IP removed from Symantec's blocklist, we have to contact the recipients of the blocked emails, and then work through them to effect a removal. Of course, these individuals are unlikely to have any idea how their organisation's spam filtering service works, and may not know who to contact. So even if we do eventually succeed getting removed from the Symantec cloud blocklist, it is likely to take considerable time and effort. Meanwhile legitimate email sent from our mail server to these organisations is being blocked, hurting both our users and the intended recipients (since our email users are their customers).

This is a good example of how you should not run an email blocklist. Providing an invalid URL in an error response is not very professional, and requiring the involvement of email end users to process a delisting request is just not acceptable. Symantec cloud customers should be aware that the inflexible way that Symantec has implemented the spam filtering service could well be preventing legitimate email from getting through, possibly with loss of business as a consequence.

False positives are unavoidable with spam filtering, but clearly dumb blocklist management is not in the "unavoidable" category. What were these guys thinking when they implemented this?