Nginx, ModSecurity, and Project HoneyPot (now with “Stop Spammers” plugin by Bryan Hadaway)

Gridvision.net is built on a foundation of Ubuntu Linux, Nginx, and ModSecurity, with WordPress as our content management system, and with Project Honey Pot providing some clever blacklist features in exchange for using our site to identify and catalog evil spambots.

Update: as of Feb 2019, we have added to our perimeter system with the outstanding WordPress plugin “Stop Spammers” by Bryan Hadaway. 

Other than a few details we prefer to keep to ourselves, this is how we did it:

Environment Background

Our core is Ubuntu 18.04 on a minimalist server we spun up at Digital Ocean.  We locked the server down with UFW immediately upon boot, and made some dramatic changes to our SSH configuration which we prefer not to detail here. Without giving away too much, we recommend sys-admin usernames that are random strings rather than any Proper Names or anything similar to them, and totally bolt down your SSH services.  It turns out our SSH server gets attacked every few seconds while our web-server receives an attack “only” every few minutes.

We use Nginx instead of Apache, and we’ve found both the software and the Nginx team to be user friendly, advanced, and cutting edge.  When Nginx put out the document, “ModSecurity 3.0 and NGINX: Quick Start Guide” on how to incorporate ModSecurity into the Nginx way of doing things, we jumped in without reservation.   You can find the Nginx tutorial at this link.  Use this link to a treasure trove of How-To documentation regarding Nginx/ModSec/Project HoneyPot all in one location.

Compiling the Shared Object Libraries

Our server couldn’t handle the compile demands of building the ModSecurity shared object library, so we had to compile on a separate machine with slightly more RAM.  This required matching our server version of Nginx with the one to be used in compile efforts, and also matching the machine architectures.  We used the opportunity to upgrade Nginx on our production machine, and were pleasantly surprised at how easy it was.  Backup first though.  Particularly config files.  As both machines were spun up at Digital Ocean, and nearly identical, we didn’t need to cross-compile.  Everything went smoothly with a bit more RAM.  Note that Nginx Plus is a paid version of the Nginx server software which gives users the option of installing pre-built binaries if compiling isn’t something you want to get into.

Note down at the bottom we’ve provided a bash script for use on your development machine. If you upgrade Nginx on your dev machine and then run the script it will automatically compile the linker library for you. This assumes you’ve taken the time to fetch and compile the libmodsecurity library (which we also show you how to do).

Once the ModSec shared-object library compiled, we rsync’d the library and its various files/directories into our /usr/local directory in production.  We then followed Nginx instructions for compiling and positioning the Nginx/ModSec Connector Dynamic Module that get the two working together. 

After a few quick [nginx -t] and [nginx -s reload] commands we had the entire setup active and running. 

Within the Nginx instructions was info on Project Honey Pot.  What a marvelous set of tools!  And the administrators over there are to be commended for creating work for the Greater Good of all Mankind.

Tweaking ModSec

Christian Folini’s tutorials on tweaking ModSec with Nginx are here.  He succinctly explains storage directories, debug modes, log entries, and fundamental operations of ModSec. From his notes, we quickly found the crs_exclusions_wordpress=1 setting, and activated it for our WordPress site.  There is a similar catchall (catchmost) setting for you Drupal users out there.  Everything needed to operate ModSec is in Christian’s tutorials.

At this point, we created an account at Project Honey Pot, received our personalized honeypot script, and “blacklist” code, and went to work on this most recent piece of kit in our fight against the evil demons of the net and their spambots.

Using Christian Folini’s notes, we learned how to identify the rules causing our honeypot activation to fail, and we made the appropriate adjustments.  ModSec has about 200 rules in their core ruleset, and one of those was catching the honeypot script.  (Identify the rule in the ModSec logs that is triggering when your honeypot activation attempt fails, and follow Folini’s notes on how to modify or turn off that specific rule).  If you configured according to the Nginx ModSec document mentioned above, you will be modifying your ModSec config file located in /etc/nginx/modsec/.  We also tweaked a few settings that were causing our online e-commerce site to incur false alarm security blocks. 

After those few rule changes were made, we then stumbled upon the final piece of the pie:  The WordPress HoneyPot Toolkit by Jeff Sterup.  What a perfect piece of work to round out our configuration!

Nginx provides excellent instructions for getting Project HoneyPot’s blacklist tools integrated into one’s site, but the plugin by Jeff Sterup is so thorough, we made the exception of installing it rather than writing our own code.  It is a very nice piece of work, and includes features that every WordPress site can benefit from.  Open your Project Honey Pot account first, then install Sterup’s plugin.

Without going into a lot of our specific settings, this short overview is how we harnessed the tools of world-class programmers who have written code for fighting the Good Fight, and made it available to the world.  If I’ve had any reservations about the future of mankind, it is the people and organizations mentioned above that give me a pinch of optimism in what has become a rather cynical, if not downright cranky, outlook.  Special thanks to the Admins at Project Honey Pot for getting us through the project.

Update Feb 2019: We started getting a lot of new user registrations with normal sounding usernames and email addresses, but something seemed a bit suspicious about them. For starters, their IP addresses all mapped to TOR exit nodes. We searched a few of the email addresses and discovered they were being used for spam. After a bit of research, we discovered an existing Word Press plugin that is just incredible. It is called Stop Spammers, and it is maintained by Bryan Hadaway.

The plugin has a lot of bells and whistles, and it comes with excellent usage tips on each setting. We found it to be immediately effective in stopping unwanted user registrations. Further testing using VPNs to camouflage our IP address didn’t cause any false blocks or failures that might cause us to lose sales or customers. We can’t say enough good things about the plugin.

Bash script for automating the compiling of nginx to make the ngx_http_modsecurity_module.so

this assumes your present working directory is ~/ which is the same as /home/your-username/.

Required Packages:
$apt-get install -y apt-utils autoconf automake build-essential git libcurl4-openssl-dev libgeoip-dev liblmdb-dev libpcre++-dev libtool libxml2-dev libyajl-dev pkgconf wget zlib1g-devxt:

Next: you will need to fetch and make libmodsecurity. Just do this occasionally. you don’t need to do this every time Nginx updates.

to fetch and compile libmodsecurity:
$git clone –depth 1 -b v3/master –single-branch https://github.com/SpiderLabs/ModSecurity

(Above is all one command. It may appear to be more than one line, but it isn’t).

then:

$cd ModSecurity
$git submodule init
$git submodule update
$./build.sh
$./configure
$make
$sudo make install

Now you have to fetch the connector library:
$cd ..
$git clone –depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git

Those steps are behind you, and you don’t need to do them often. Below is what you need to do every time Nginx upgrades to a newer version:

If you followed my instructions above carefully, all the various files and libraries you need have been built in your home directory. Now, we are going to run a script from inside home that determines your version of Nginx, and then compiles the ngx_http_modsecurity_module.so library. Ready? Here you go.

#!/bin/bash

#nginx -v 2>&1 | tee version.txt

a=$(nginx -v 2>&1)

echo 'test'
echo "$a"
echo 'test2'


b=${a#*/}
echo "$b"
wget_string="wget http://nginx.org/download/nginx-$b.tar.gz"

echo "$wget_string"
echo "launching wget operation"
$wget_string

untar_string="tar zxvf nginx-$b.tar.gz"
$untar_string

cd_string="cd nginx-$b"
$cd_string
./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
make modules

cp ~/nginx-$b/objs/ngx_http_modsecurity_module.so ~/

echo "Done with the building of ngx_http_modsecurity_module.so"
echo "Now about to scp it up to me@production-server:/home/my-username/"

scp ~/nginx-$b/objs/ngx_http_modsecurity_module.so me@production-server.com:/home/my-username/

That bash script will save you untold amounts of frustration when you discover that Nginx has been upgraded to a new version. Just change the scp command to fit your needs if you use it to scp up to a production environment. Otherwise, you don’t need the scp command.

We also note that Nginx has changed the location of where this all-important library needs to go:

cp ngx_http_modsecurity_module.so /user/lib/nginx/modules/

the /etc/nginx/modules directory is now just a link. The actual library files reside in /user/lib/nginx/modules.

You are welcome!