During the last few months, I have become a system administrator for two dedicated web servers. One of them is hosting ~100 accounts, and one of the smallest accounts got onto the “bad list” of some eastern block spammer last weekend. The account owners don’t even have their MX record pointed at our server – we aren’t supposed to see any mail for them.
On this past Monday alone we received over 1 million emails from over 31,000 unique IP addresses. Yes, it was all for this one account. After sending “their mail” into a black hole, our server was still unusable. All of the mail lookups, log writing, &c. continued to take a toll. The company that houses the server added an additional iptables rule:
iptables -I INPUT -m string --algo bm --string "@example.com" -p tcp --dport 25 -j REJECT --reject-with tcp-reset -m comment --comment "STMP Block"
I had never used the “string” module for iptables but have seen it referenced before. This command basically causes iptables to search every packet on port 25 for the string “@example.com” using an efficient search algorithm. When it finds a match, it drops the connection. Great idea.
There are a couple of weaknesses to this method:
-
Iptables only notifies the remote computer that it has closed the connection. Our local Exim process thinks the other server is still connected until we hit the timeout threshold. With limited memory, and Apache processes, we want to limit how many Exim processes get started.
-
There is nothing to stop the bot from coming back (~32 emails per IP in a 24 hour period).
There are programs designed for this scenario, such as fail2ban. Its built-in scripting should handle this well but, being me, I wanted to find something with a smaller footprint (especially since we were already under attack).
Several years ago I had kept a publicly-available server at my house but hid it behind a “port knocking” script. You had to hit three specific ports in a specific order to gain access. Iptables had been flexible enough to support that so I merged that idea with the command above to create a reusable script:
IPTABLES=/usr/sbin/iptablesn
STRING='example.com'
PORT=25
# Check new connections to ${PORT} against the blacklist.
$IPTABLES -A INPUT -p tcp --dport ${PORT} -m recent --name blacklist --update --seconds 86400 -j DROP
# When blackchain is triggered, add IP to blacklist and drop the current connection.
$IPTABLES -N blackchain
$IPTABLES -A blackchain -m recent --set --name blacklist -p tcp -j REJECT --reject-with tcp-reset
# Trigger the entry of our BLACKCAIN chain...
$IPTABLES -A INPUT -p tcp --dport ${PORT} -m string --string ${STRING} --algo bm -j blackchain
It creates a “blacklist” of the most recent IP addresses that have sent an email to “@example.com” on port 25. Anybody who attempts this will be dropped immediately and prevented from reconnecting to port 25 for 24 hours.
There is just one tiny problem. The “recent” module only remembers the last 100 IP addresses by default. This can be remedied one of two ways.
The first method might be difficult to pull off when your firewall is running:
rmmod xt_recentn
modprobe xt_recent ip_list_tot=5000
Thankfully there is an alternate method that is available on some systems:
chmod +w /sys/module/ipt_recent/parameters/ip_list_totn
echo 5000 > /sys/module/ipt_recent/parameters/ip_list_tot
Make sure you don’t adjust this value rashly. I had tried 36,000 and that caused problems. At the moment I have our list up to 10,000 entries and the server seems to be handling the number without issue.
You can view your list at /proc/net/ipt_recent/blacklist. wc -l can be used to determine how full your list is.
Happy anti-(?) hacking!