You are a spammer and you don’t even know it!

Over the past several months I have been seeing an increase in the number of customer accounts being abused to relay spam via our server. In each case the customer account credentials have been used to authenticate to our SMTP service and spam.

Do I think these customers have decided to get into the “business” of sending spam? No.

I do, however, have to behave in pretty much the same way as if I did which means in each case the customer account is immediately terminated.

Previously I have tried other approaches. I have changes the password on the account (and on all associated accounts and email only accounts, etc) and contacted the customer to let them know and ask them to fix their security. The problem with this is that in almost all cases the customer denies that they are at fault or is incapable of fixing their security properly so the problem happens again fairly quickly with the same account.

It may seem that terminating the customer account is too harsh but reality is that when a customer account is abused in this way it damages the whole service. Other email servers refuse to accept any email from our server and that harms all customers. In order to remedy this I have to spend significant amounts of time cleaning up our email queues and working with other ISPs and blocklist operators to get the UKFSN outbound email service working again.

This weekend I had to terminate two customer accounts because of this. One begins “dab” and the other “chr”.

Posted in Email, SPAM

Outage Yesterday

Yesterday afternoon UKFSN suffered a serious outage our main servers. This knocked out all web servers, the main email servers and the RADIUS authentication service.

Broadband, Backup MX service and DNS services were not affected.

The loss of service started at approximated 4pm. Unfortunately I was offline at the time and did not discover the problem until nearly 7pm. Attempts to restore service remotely were not successful and ultimately it was necessary for an engineer at the Hetzner DC to physically power cycle equipment to get the servers back online which was done by about 9pm.

This brought the servers back online however it was still necessary to repair filesystems before services could be fully restored. Between 10:30pm and 11:30pm all services were brought back online.

Please accept my apology for any inconvenience this loss of service may have caused you.

Posted in Uncategorized

Filtered Internet Access

You may have seen an announcement today by David Cameron, the British Prime Minister, in which he states that all ISPs in the UK have agreed to block pornography (and other materials) by default with customers being required to deliberately state that they wish to access such materials in order to get them.

The announcement is essentially an enormous lie. Some ISPs may have agreed to do this but by no means all and moreover what the announcement claims will happen is simply not possible.

Certainly some of these materials can be blocked but the unfortunate truth is that no systems can block all of them and that any system that is successful to any significant degree in getting near to doing so will also end up blocking perfectly innocent sites.

UKFSN has no agreed to implement any such block. Unless it is absolutely required by specific legislation UKFSN will not implement any such block.

Posted in Uncategorized

Problems with AWBS on Debian Stable

Recently I moved the UKFSN services to a new platform which involved migrating everything to new servers. UKFSN runs on Debian Linux and had been running on an older version of the Debian stable distribution. The new servers are all running the latest version of Debian stable.

A significant consequence of this upgrade is that the version of PHP on our servers went up to 5.4 and that caused many problems. A particular problem arose with the software we use to give customers direct control of domain name registration and management which is called AWBS. This software package is written in PHP and it simply does not work with version 5.4 of PHP.

Version 5.4 of PHP was released on 1st March 2012. This release contained many changes but the really important ones were the removed of deprecated features. That means features the authors of PHP had stated a long time ago would be removed and should not be used. AWBS was dependant on one of those features to the extent that important bits of it simply stopped working.

After investigation I was able to identify the source of the problem and ascertain that it is not possible to fix it – I really should not be using software for UKFSN that is not released under a free software / open source license but there isn’t anything readily available under such a license that does what I need for UKFSN. In the short term the solution has been to downgrade the version of PHP to 5.3.

Long term the solution is to finish the work started some time ago to write a completely new ISP management for UKFSN and just get rid of AWBS.

Posted in Uncategorized

How to test sending email using MIME::Lite

Earlier this week I had to update a system that uses the MIME::Lite module to send email so that the emails it sends are properly UTF-8 encoded. The change to the code was very simple.

What was a little less simple was writing a test to verify that email is properly encoded. Note that I stated that I was writing a test. When developing software it is not enough to manually test that something does what is expected. We write tests so that we can repeatedly prove that our systems work as they are expected to and also so that we can identify if a later change breaks something.

Thankfully I was programming in Perl which means I had a flexible language to work with. The solution was to redefine MIME::Lite’s send method like so:

no warnings "redefine";
*MIME::Lite::send = *MIME::Lite::as_string;
use warnings "redefine";

What this does is firstly to turn off warnings for redefines. If you use warnings in your Perl programming you will get a runtime warning when you redefine a subroutine or method and we don’t want that warning as we fully intend to do this. Do note that the warnings on redefine are re-enabled immediately once we’re done though.

After we’ve ensured that Perl wont raise a warning we go ahead and redefine the send() method. In this case we redefine it to MIME::Lite’s own as_string method which simply returns the entire email, headers and body, as a single string. The code which calls send() captures the return value and sends it up the calling stack so we will get this back. Even though the code expects to be sending back true or false the fact that Perl doesn’t do strict variable typing means we can do this safely. You can redefine to anything. It’s fairly common to redefine to something as simple as sub { return $_[0] } but in this case MIME::Lite has as_string so we used that.

With this in place I was free to simply call higher level methods in the classes and objects which were in turn using MIME::Lite objects to send email and safely test the resulting emails.

Posted in Code, Email, Perl, Testing

Why you should learn perl programming

Last weekend I attended the London Perl Workshop. One of the discussions was centred on the state of the jobs market for Perl programmers.

If you listen to many in the media and those who measure the popularity of programming languages Perl might not seem like the best choice for someone who is considering a career in programming. You might well end up considering that Java, C, Python or even PHP are where you should be focusing and certainly the first three are good bets however what you would be missing is that there is a very strong demand in the real jobs market for Perl developers.

Not only is there strong demand but pay rates are also very good as that demand is generally from successful companies. Moreover those who program in Perl benefit from the availability of a vast range of well written libraries on the CPAN which make it easier to produce high quality code that is easy to maintain. The days when you might reasonably fear that Perl programming resulted in spaghetti code not even the author could maintain are long gone. Modern perl programming produces clear code with well established standards which mean large teams are able to work well together.

I recently found myself having to dig into a very large Perl codebase (over 100,000 lines of code) which is already nearly 10 years old. One might expect that this was a painful experience but in fact that was far from the truth. The code was clear, readable and easy to update without fear of breaking things.

The fact that Perl has a fantastic testing framework (courtesy of CPAN) makes life even better as it is easy to write and maintain a suite of tests to ensure that changes to the software do exactly what is expected and do not have unexpected results.

Of course Perl is a Free Software/Open Source language so you get all the benefits of continuous development with an immediate focus on what developers and those who employ them actually need and of a dynamic and, mostly, friendly and intelligent community around the language.

UKFSN uses the LAMP stack and here the P is definitely Perl.

Posted in Uncategorized

Preventing SMTP Auth abuse

One of the biggest problems I have had to deal with over the past few years has been abuse of authenticated SMTP. Spammers occasionally manage to obtain a customer’s username and password and use these to send email via our servers with authenticated SMTP.

It’s not as easy to prevent this as it is to prevent other attempts to block spammers as we need to ensure that genuine customers are able to send email without problems. The solution is not however enormously difficult and a discussion I had today has encouraged me to publish the generic bits of it so others can use it or pick holes as appropriate.

WordPress does horrible things to the code indenting so you’ll need to forgive the horrible style.

use Tie::Hash::Expire;
use File::Tail;

if ( $> != 0 ) {
    die "You must be root to run this command";
my %seen = ();
tie %seen, 'Tie::Hash::Expire', { expire_seconds => 1200 };
my $line = undef;
my $log = File::Tail->new(name => "/var/log/mail.log", tail => -1);
while ($line = $log->read) {
    next unless $line =~ /: client=.*\[(.*)\], sasl_method=(.*), sasl_username=(.*)/;
    my $ip = $1;
    my $user = $3;
    $user =~ s/, sasl_sender=.*$//;


    my $size = keys %{$seen->{$user}};
    if ($size > 2 ) {
        # "Customer $user looks like they have been compromised\n";
        my ($l, $d) = split /@/, $user;
        if ( $d ) {
            # full email address - local part is $l and domain is $d
            # XXX Do something here to suspend the account - change password?
    else {
        # just a username. Again suspend it

This should be fairly clear. All it does is to tail the mail log file and track the sasl login and the IP address of the client connection. If the same user login connects to the authenticated SMTP service from more than 2 client IP addresses within a short time some action is taken to suspend the account.

On our real implementation of this there is a further check on the client IP address so that we don’t track connections if we recognise the IP address as belonging to a customer broadband connection.

Posted in Code, Email, Perl, Postfix, SPAM