Forensic Analysis of a Spam Attack

Recently one of the sites I host was targeted by some script kiddie who used a fairly old exploit in a WordPress theme to misuse the server for sending spam. The way this in general works is that they use a known vulnerability in the Blog or CMS software or addon you use which gives them access to the file system to upload arbitrary scripts. They then upload so called injection scripts, for example C99, or something else. This scripts can be executed from outside and can be used to upload more files, read files containing login information, query your database, or what ever is possible for them to do from that point on in your system.

This has happened to me before and it is more then annoying as this poses a threat to the mailing system I and so many others rely on. Becoming blacklisted is a real pain and a real damage. This time I took the chance and time to investigate the incident in much detail and I want to give here a overview and document the steps I followed.

Securing and Analyzing Postfix

Around 6 o’clock the day before some users noticed difficulties with their emails being delivered. A quick check of the system revealed that the queue was piled up and the delivery of mails was hindered. A quick look at mailq helps:

66.109 emails queued, which is really a lot for the system I run, and most/all of them originate from webmaster@example.com. This already makes quite clear what happened as webmaster usually indicates that the website is sending emails, so example.com got penetrated. A detailed view can help to understand better and estimate the damage happend. For this I like to use pflogsumm as well as mailgraph. If your interested in ways to aggregate postfix logs you might want to take a look at “Reliably Store Postfix Logs in S3 with Apache Flume and rsyslog”

pflogsumm shows that the main target of the attack were some of the major mail services. Most of them have their own blacklist and are really not that much fun to deal with. Understandably they also try hard to omit any harm by this kind of spam attacks.

A way you can make postfix itself more resilient is to limit the amount of concurrent delivery and shaping the traffic. This has two valuable benefits for one the targeted mail services are less likely to blacklist you and it brings incidents like this to your attention much faster, if you are not monitoring postfix anyways. To do this set the following parameters in main.cf of postfix:

This avoids hitting any target too hard at any time. Postfix does offer of course some more parameters like *_destination_recipient_limit which can be found here.

Cleaning up

What I did next is actually stop postfix  service postfix stop  as this immediately stops the spam and gives me time to prevent further damage an clean up. Alternatively or in addition you can also halt all emails in the queue by postsuper -h ALL. Next I cleaned up the queue. I wanted to remove the spam so when I would restart postfix later it will not get send. Of course I only want to delete the spam and not any email that is rightfully queued to be send by a customer. I filtered by sender an here is the command I used to remove all the queued emails originating form webmaster@example.com:

This extracts the IDs of the the queued emails and deletes them.

Next I disabled and deactivated all accounts and the website of example.com so I would be able to quickly restart postfix for all customers.

Controlling the damage it’s a good idea to check whether the server has been blacklisted. There exist some service who incorporate most of the common blacklist. This unfortunately does not include companies having their own black lists like aol. Some other lists to check:

  1. http://mxtoolbox.com/blacklists.aspx
  2. http://whatismyipaddress.com/blacklist-check

 Investigating the Intrusion

To investigate what has happened and what security hole was used it is a good start to look at the web logs and dig out some suspicious behavior. If you know the site well it could already be enough to just browse through the folder and look for unknown files and/or go through files recently changed.

If you know which files were used for spam or you can narrow down the time interval the intrusion occurred it helps you to go through the logs.

By browsing through the files I found some suspicious files. In my case it was named  plato.php  and was located under  wp-content  (the site was based on WordPress). The file contained something similar to this:

This seems to be a common way to inject code into a server and deflating and decoding the content revealed that the first intuition was right. This is the code of a so called injection script named C99. The header says it all:

I drilled a little deeper and found some more files which also contained the signature of the script kiddie that might be responsible for the attack. A young boy from Jakarta/Indonesia who is naming himself Vito RawckerheaD and makes some quick bugs selling and probably also using tools to spam – http://vitoshell.com/ | https://twitter.com/vito_DarkHackeR.

But how did he gain access to the system? As I now knew the files used by the attacker I was able to go through the logs and find the first call of plato.php. With that I had the IP of the attacker used and was also able to investigate the calls the IP made. Already by looking at the access logs of the IP it was noticeable that a lot of images were called. What ever the security hole was it seems to be related to images.

Having the IP it is also nice to know from where the attack was triggered.

So the attack itself also originated from Indonesia the Country our famous author of the mailer is from.

Still we don’t know how the access was gained. Using the IP to through the error.log revealed nothing new but the first log entry before the first call to plato.php gives a clue.

The file  external_165157e54fca14624738a762ee85353a.php  also contains some eval code and seems to be the entry point to the system. Searching the Internet for timthumb.php contains a ZeroDay exploit (1) which exposes the file system to the outside. The IP of that initial request was not from Indonesia but leads to brainhost.

To  clean up the files compromised I listed all the files date changed and found some “hidden” files containing the injection code. To list all files by date:

Keeping WordPress Secure

So a plugin in a WordPress theme was responsible for the security vulnerability. It’s is crucial to regularly update your WordPress installation and the plugins you use. Also keep your self up to date to the most recent threats found in plugins and WordPress itself. Some sources I found to help are:

  1. The Definite Guide to WordPress Security
  2. Wordfence

Conclusion

This weekend I actually wanted to blog about something else but was completely occupied by this spam attack to my system. I tried here to document the steps I did to drill down to the problem and find the source of the attack. I am still not sure if I was able to discover all of the files and database entries effected and the site is still closed. It will still take some time to clean up completely and re-establish the site. On the bright side I learned a lot and the most valuable lesson of all is probably that I need to invest more in to appropriate monitoring. This will lead to some more interesting project I’ll sure blog about here.

Futher Reading

  1. WordPress 3: Das umfassende Handbuch. Aktuell zu Version 3.8 (Galileo Computing) (Amazon)
  2. WordPress Security Fundamentals: Protect Your Website from Hackers and Identify WordPress Security Issues (Amazon)
  3. WordPress Security: Kugel-sichere WordPress Installation im Handumdrehen (Amazon)

2 thoughts on “Forensic Analysis of a Spam Attack”

Leave a Reply

Your email address will not be published. Required fields are marked *