Recently one of the sites I host was targeted by some script kiddie who used a fairly old exploit in a WordPress theme to misuse the server for sending spam. The way this in general works is that they use a known vulnerability in the Blog or CMS software or addon you use which gives them access to the file system to upload arbitrary scripts. They then upload so called injection scripts, for example C99, or something else. This scripts can be executed from outside and can be used to upload more files, read files containing login information, query your database, or what ever is possible for them to do from that point on in your system.
This has happened to me before and it is more then annoying as this poses a threat to the mailing system I and so many others rely on. Becoming blacklisted is a real pain and a real damage. This time I took the chance and time to investigate the incident in much detail and I want to give here a overview and document the steps I followed.
Securing and Analyzing Postfix
Around 6 o’clock the day before some users noticed difficulties with their emails being delivered. A quick check of the system revealed that the queue was piled up and the delivery of mails was hindered. A quick look at mailq helps:
...
s5A13ixI026629 27689 Tue Jun 10 03:03 <webmaster@example.com>
(Deferred: Connection timed out with interstat.com.)
s5A144kY026937 32682 Tue Jun 10 03:04 <webmaster@example.com>
(Deferred: Connection refused by mail.capital-otels.com.)
s5A15URP028725 44767 Tue Jun 10 03:05 <webmaster@example.com>
(Deferred: Connection refused by mailme.test.com.)
Total requests: 66109
66.109 emails queued, which is really a lot for the system I run, and most/all of them originate from webmaster@example.com. This already makes quite clear what happened as webmaster usually indicates that the website is sending emails, so example.com got penetrated. A detailed view can help to understand better and estimate the damage happend. For this I like to use pflogsumm as well as mailgraph. If your interested in ways to aggregate postfix logs you might want to take a look at “Reliably Store Postfix Logs in S3 with Apache Flume and rsyslog”
$ pflogsumm -d today /var/log/mail.log
... Host/Domain Summary: Message Delivery -------------------------------------- sent cnt bytes defers avg dly max dly host/domain -------- ------- ------- ------- ------- ----------- 27884 60767k 5 36.2 m 4.6 h gmail.com 14091 29778k 0 40.6 m 3.2 h hotmail.com 12649 26959k 2917 33.9 m 2.4 h webmail.co.za 10465 19981k 33783 29.4 m 8.3 h yahoo.com ...
pflogsumm shows that the main target of the attack were some of the major mail services. Most of them have their own blacklist and are really not that much fun to deal with. Understandably they also try hard to omit any harm by this kind of spam attacks.
A way you can make postfix itself more resilient is to limit the amount of concurrent delivery and shaping the traffic. This has two valuable benefits for one the targeted mail services are less likely to blacklist you and it brings incidents like this to your attention much faster, if you are not monitoring postfix anyways. To do this set the following parameters in main.cf of postfix:
smtp_destination_concurrency_limit = 2 smtp_destination_rate_delay = 1s smtp_extra_recipient_limit = 10
This avoids hitting any target too hard at any time. Postfix does offer of course some more parameters like *_destination_recipient_limit which can be found here.
Cleaning up
What I did next is actually stop postfix service postfix stop as this immediately stops the spam and gives me time to prevent further damage an clean up. Alternatively or in addition you can also halt all emails in the queue by postsuper -h ALL. Next I cleaned up the queue. I wanted to remove the spam so when I would restart postfix later it will not get send. Of course I only want to delete the spam and not any email that is rightfully queued to be send by a customer. I filtered by sender an here is the command I used to remove all the queued emails originating form webmaster@example.com:
mailq | fgrep webmaster@example.com | sed 's/*.*//' | awk '$0 {print $1}' | tr -d '*!' | postsuper -d -
This extracts the IDs of the the queued emails and deletes them.
Next I disabled and deactivated all accounts and the website of example.com so I would be able to quickly restart postfix for all customers.
Controlling the damage it’s a good idea to check whether the server has been blacklisted. There exist some service who incorporate most of the common blacklist. This unfortunately does not include companies having their own black lists like aol. Some other lists to check:
Investigating the Intrusion
To investigate what has happened and what security hole was used it is a good start to look at the web logs and dig out some suspicious behavior. If you know the site well it could already be enough to just browse through the folder and look for unknown files and/or go through files recently changed.
find /var/www -type f -printf '%TY-%Tm-%Td %TT %pn' | sort
If you know which files were used for spam or you can narrow down the time interval the intrusion occurred it helps you to go through the logs.
By browsing through the files I found some suspicious files. In my case it was named plato.php and was located under wp-content (the site was based on WordPress). The file contained something similar to this:
<?php
eval(gzinflate(base64_decode('7P1pe+O4sQAKf8/....')));
?>
This seems to be a common way to inject code into a server and deflating and decoding the content revealed that the first intuition was right. This is the code of a so called injection script named C99. The header says it all:
/*******************************************/ /* c99 injektor v1 06.2008 */ /* Re-coded and modified By PLaTo */ /* #r@ Crew */ /*******************************************/
I drilled a little deeper and found some more files which also contained the signature of the script kiddie that might be responsible for the attack. A young boy from Jakarta/Indonesia who is naming himself Vito RawckerheaD and makes some quick bugs selling and probably also using tools to spam – http://vitoshell.com/ | https://twitter.com/vito_DarkHackeR.
But how did he gain access to the system? As I now knew the files used by the attacker I was able to go through the logs and find the first call of plato.php. With that I had the IP of the attacker used and was also able to investigate the calls the IP made. Already by looking at the access logs of the IP it was noticeable that a lot of images were called. What ever the security hole was it seems to be related to images.
# cat *access.log | grep plato.php | head 118.96.228.153 - - [13/Jun/2014:08:38:16 +0200] "GET /wp-content/themes/arras/library/cache/plato.php HTTP/1.1" 200 5983 "-" "Mozilla/5.0 (Windows NT 6.1; rv:30.0) Gecko/20100101 Firefox/30.0" 118.96.228.153 - - [13/Jun/2014:08:38:22 +0200] "GET /wp-content/themes/arras/library/cache/plato.php?act=img&img=home HTTP/1.1" 200 209 "http://www.torrox.de/wp-content/themes/arras/library/cache/plato.php" "Mozilla/5.0 (Windows NT 6.1; rv:30.0) Gecko/20100101 Firefox/30.0"
Having the IP it is also nice to know from where the attack was triggered.
# whois 118.96.228.153 % [whois.apnic.net] % Whois data copyright terms http://www.apnic.net/db/dbcopyright.html % Information related to '118.96.192.0 - 118.96.239.255' inetnum: 118.96.192.0 - 118.96.239.255 netname: TLKM_D2_BB_SPEEDY_CRB country: ID descr: PT TELKOM INDONESIA descr: Cirebon, Jawa Barat admin-c: AR165-AP tech-c: AI64-AP remarks: ------------------------------------------------------------------ remarks: Broadband Service for Cirebon (Jawa Barat). remarks: ** These IP was used dinamically for end user. ** remarks: Send ABUSE and SPAM reports with plain ASCII text only to remarks: to abuse@telkom.net.id. remarks: The netname enclosed in square bracket is included in the subject. remarks: ------------------------------------------------------------------ status: ASSIGNED NON-PORTABLE changed: hostmaster@telkom.net.id 20060501 mnt-by: MAINT-TELKOMNET source: APNIC role: PT Telkom Indonesia ABUSE INTERNET Response Team address: PT. TELKOM INDONESIA address: Menara Multimedia Lt. 7 address: Jl. Kebonsirih No.12 address: JAKARTA country: ID phone: +62-21-3860500 fax-no: +62-21-3861215 e-mail: abuse@telkom.net.id admin-c: HM444-AP tech-c: HM444-AP nic-hdl: AI64-AP notify: hostmaster@telkom.net.id mnt-by: MAINT-TELKOMNETS changed: hostmaster@telkom.net.id 20060105 source: APNIC
So the attack itself also originated from Indonesia the Country our famous author of the mailer is from.
Still we don’t know how the access was gained. Using the IP to through the error.log revealed nothing new but the first log entry before the first call to plato.php gives a clue.
[Fri Jun 13 08:36:55 2014] [error] [client 192.64.181.136] PHP Warning: imagecreatefromgif(): '/wp-content/themes/arras/library/cache/external_165157e54fca14624738a762ee85353a.php' is not a valid GIF file in wp-content/themes/arras/library/timthumb.php on line 371
The file external_165157e54fca14624738a762ee85353a.php also contains some eval code and seems to be the entry point to the system. Searching the Internet for timthumb.php contains a ZeroDay exploit (1) which exposes the file system to the outside. The IP of that initial request was not from Indonesia but leads to brainhost.
To clean up the files compromised I listed all the files date changed and found some “hidden” files containing the injection code. To list all files by date:
find $1 -type f -exec stat --format '%Y :%y %n' {} ; | sort -nr | cut -d: -f2- | head
Keeping WordPress Secure
So a plugin in a WordPress theme was responsible for the security vulnerability. It’s is crucial to regularly update your WordPress installation and the plugins you use. Also keep your self up to date to the most recent threats found in plugins and WordPress itself. Some sources I found to help are:
Conclusion
This weekend I actually wanted to blog about something else but was completely occupied by this spam attack to my system. I tried here to document the steps I did to drill down to the problem and find the source of the attack. I am still not sure if I was able to discover all of the files and database entries effected and the site is still closed. It will still take some time to clean up completely and re-establish the site. On the bright side I learned a lot and the most valuable lesson of all is probably that I need to invest more in to appropriate monitoring. This will lead to some more interesting project I’ll sure blog about here.
henning.kropponline.de 
Forensic Analysis of a Spam Attack http://t.co/JAoyL9HmKD
LikeLike
Von @jonbros: Forensic Analysis of a Spam Attack http://t.co/P7zGgVf7HP #IronBloggerMUC
LikeLike