Friday, December 27, 2013

WordPress Spammer Mitigation

If you run servers that host WordPress sites, no doubt you're eventually going to run into issues with resource consumption due to spammers.  There are a lot of great mitigation options, including:

  • CAPTCHA plugins
  • Anti-spam plugins (like Akismet)
  • Security plugins or configurations (black/whitelists, .htaccess rules, etc)
These are all great, and I encourage you to look into them, but for the scope of this post I'm going to go hardball and just straight up block whole subnets using some Apache log analysis and iptables rules.  You're probably not going to want to do anything this extreme, but bits of this may be useful to you.

One of the sites that I run servers for at $WORK has been getting pounded by comment spammers over the last few weeks.  This hasn't been much of a problem for the webservers, but the database server has been under some really high load to keep up with all the transactions.  Knowing the site pretty well, I can make a few assumptions.

The Assumptions: 
  • The largest portion of our traffic is from internal IPs
  • Most IPs won't be commenting more than a handful of times in 24 hours
Using these assumptions, I can make a very rough educated guess about each IP that shows up in the logs.

In order to post a comment, an IP will submit an HTTP POST to "wp-comments-post.php".  That script has nothing there if you browse to it, so there aren't any legit HTTP GET requests.  Any time "wp-comments-post.php" shows up in the logs, it's going to be a comment.  Using this, we can get a list of everyone who's posted a comment.

# Search for wp-comments-post.php; print the IP of the user

$ awk '/wp-comments-post.php/ {print $1}' <your_apache_access_log(s)>


36.248.168.176
36.248.168.176
175.44.56.169
198.27.83.88
112.111.173.43
61.154.153.155
36.248.168.176
192.184.52.146
(...)


That's kind of useful, but let's also sort the IPs and count how many times they've posted.

# Same as before, but sorted by number of occurrences of an IP

$ awk '/wp-comments-post.php/ {print $1}' <your_apache_access_log(s)> | sort | uniq -c


      2 77.231.117.122
      2 77.247.181.162
     22 79.100.50.24
      2 79.114.121.253
      1 79.167.52.220
      2 80.79.120.247
      1 80.79.121.242
      3 80.82.64.242
(...)

So already we have a likely suspect, given that it's only noon and they've commented 22 times from one IP.  But it's borderline.  79.100.50.24 there will have to get a lot worse for me to count it.  We can handle 22 comments.

However, look at how close some of those subnets are.  Sure, there are thousands of IPs in each, but given that most of our traffic will be coming from internal IPs, that's sort of suspicious.  And looking at the full list, there are thousands of IPs that are really close together and they're all commenting.  Suspicious.  Spammers are sneaky.  We can, though, update our assumption list.

The (New) Assumptions: 
  • The largest portion of our traffic is from internal IPs
  • Most IPs won't be commenting more than a handful of times in 24 hours
  • ...which means most external /24 subnets won't comment more than a few dozen times in 24 hours
So, let's set the bar high and say if an external /24 subnet writes more than 200 comments in 24 hours, they're on the list of suspected spammers.  We can find these guys.

# Search for wp-comments-post.php 
# and break the IPs into /24 subnets instead

$ awk '/wp-comments-post.php/ {print $1}' <your_apache_access_log(s)> |awk -F. '{print $1"."$2"."$3".0/24"}'

The second awk command there breaks the output into fields by using a period (.) as the separator (the "-F.") and then prints the first three octets and a 0/24 to represent it's subnet.  

From there, let's sort the subnets, count the number of occurrences, and sort again by the volume of occurrences:

# Break up comments into subnets 
# and sort by the number of comments per subnet

$ awk '/wp-comments-post.php/ {print $1}' <your_apache_access_log(s)> |awk -F. '{print $1"."$2"."$3".0/24"}' '| sort | uniq -c | sort -nb -k1  

The second sort does the sort on the first column (-k1) and interprets them numerically (the "n" in -nb) and ignores leading blanks (the "b" in -nb).  This is my new output:

(...)
     81 110.89.61.0/24
     84 198.27.67.0/24
    127 120.37.210.0/24
    170 124.238.18.0/24
    172 27.156.77.0/24
    198 59.38.138.0/24
    235 36.248.168.0/24
    549 175.44.56.0/24

Looks like we have some likely suspects.  79.100.50.24's subnet didn't make the list of the top 8 - must have been borderline enough.  I'll go ahead and block those top three (seriously - 549 comments?) and consider the others borderline.  Our other spam mitigation techniques can handle them if they're spamming and not legit.

We're running puppet on all our servers, and I've included a module that will take IPs in an array and add a rule for each in the iptables for all the servers, so these guys get banned from this server, and all the others we're running as well:

REJECT     tcp  --   175.44.56.0/24      0.0.0.0/0           tcp /* block-spammers */ reject-with icmp-port-unreachable 

This is just an example of parsing logs and acting on the data we can obtain from them, and it's admittedly an extremely heavy-handed approach.  You should, of course, tailor this to your own environment.  If you're running a popular site with tons of legit comments, using this as-is would be a very bad move.






SSL Key and Certificate Matching

Symptoms:

  • Apache fails to start or restart
  • No errors on STDERR
  • No errors in logs
  • You've recently changed an SSL certificate or the SSL config for a Virtual Host

Diagnosis:

When this happens to me, 99% of the time my SSL key and certificate do not match for some reason (old key with new cert, copy error, vhost typo, etc).  Apache is really not helpful when this occurs. 

"Hey, $JUNIORADMIN!  I'm just not going to start.  Oh, you want error logs?  No, I think I'll skip that.  An error on STDERR?  Nope.  None of that either.  In fact, I think I'll just sit here doing nothing and silently mocking you."

Thanks Apache.

Check that your key and certificate match.  You can do this by comparing the modulus for each of them to see if it is a match.  To do this with openssl:

# Check the Key
$ openssl rsa -noout -modulus -in <your_ssl_key_file>


Modulus=B365389CC131D10073BE0F017C25F095EF0C4A79A8F8881943CB960F0C2AE4D7B4AAE199AB8CA6DC7735FD8919AEF9904C92196CF277ADBC7798AA7D7B479B2923ADA4A8E10B724F8227EFD431A1F9D6ED2799B1C68DA4974D8E6EF48DFC73BE9EBE79A2B51D94B97E5CF52CC8DAA2AF24749D4A33098B92DD1896A9306C2F883CBB17F1F148A57DC3525D453D4C65D93EA8F353737406106412D4A523D1A137BB9BBB5589B3B1737687F68185F856BC9A323B8E93A8BAB1755B47A8724D9CE5FB2CE323371DAF279A27867188C65EE1356F0C867189620A932726B2909D7777CD14C0EB8E3956BA43014EB2D94BC5FC2F64C69152354DFDBA3BB9AC9743894D


# Check the Cert
$ openssl x509 -noout -modulus -in <your_ssl_cert_file>

Modulus=B365389CC131D10073BE0F017C25F095EF0C4A79A8F8881943CB960F0C2AE4D7B4AAE199AB8CA6DC7735FD8919AEF9904C92196CF277ADBC7798AA7D7B479B2923ADA4A8E10B724F8227EFD431A1F9D6ED2799B1C68DA4974D8E6EF48DFC73BE9EBE79A2B51D94B97E5CF52CC8DAA2AF24749D4A33098B92DD1896A9306C2F883CBB17F1F148A57DC3525D453D4C65D93EA8F353737406106412D4A523D1A137BB9BBB5589B3B1737687F68185F856BC9A323B8E93A8BAB1755B47A8724D9CE5FB2CE323371DAF279A27867188C65EE1356F0C867189620A932726B2909D7777CD14C0EB8E3956BA43014EB2D94BC5FC2F64C69152354DFDBA3BB9AC9743894D

That's kind of tedious, and I've gotten into the habit of checking them every time I make a change to the key and certificate, so I found it easier to make a quick bash script to do this for me.

#!/bin/bash
KEY=$1
CRT=$2
KEY_MOD=$(openssl rsa -noout -modulus -in $KEY)
CRT_MOD=$(openssl x509 -noout -modulus -in $CRT)

if [ "$KEY_MOD" != "$CRT_MOD" ] ; then
  echo "No Match"
  exit 1
else
  echo "Key and Certificate match"
fi

Pass the script two arguments, first the key, then the certificate, and it'll compare the two strings for you.