✓ Solved

Traffic spike alerts

Hello! I'd love some feedback and suggestions on a weird issue we are having. For the last several weeks, we have been getting alert emails about a few of our servers having inbound traffic spikes and outbound traffic spikes to our database box. I looked into the access logs the first week and there were 3 IP's that were hitting the box waaaay more than any others. I created a firewall entry and blocked them. While it wasn't a for sure solution, it did slow them down. Now it's spiking back up with different IP's.

I have seen quite a few forum posts about iptables but that's only for static IP's right? How can I keep these people from just changing IPs and continuing to hit the box?

Thanks!!

2 Replies

✓ Best Answer

Within iptables there are two ways to do this. Actually there are more ways but the following two are the most common ways and you'll find examples of each using a search engine.

  1. Connlimit. The way connlimit works in a nutshell is after an ip opens multiple connections, we'll say 6 connections, it blocks all future and current connections from that ip until that ip stops sending traffic entirely and timeouts out of kernel conntrack (which is 180 seconds iirc but that time might not be correct).

So too many connections= temp ban.

  1. (I put 2 here. weird bug?). Anyways: Hashlimit. This is similar to Hitcount but better and isn't limited to just 255 packets per second like hitcount is. Hitcount however is more simple to use. One way hashlimit works is like an overall count of packets from the source ip. It can also count entire subnets. Anyways it drops packets (or bytes) from the ip if the ip sends too many in your given time interval. Either hashlimit or hitcount can filter the individual source ip dynamically, so you don't have to manually add any ips to any lists.

So both together (hashlimit, OR hitcount, AND connlimit) is what's really going to help since they can all filter dynamically without needing to know static ips beforehand. Think of it like this:

even with hashlimit doing an overall packet limit from the attacker ip, the attacker ip can still use a small number of packets to open a bunch of tcp connections. So connlimt comes in if they try that too :)

It's also worth noting that hashlimit or hitcount can filter earlier than connlimit. So rather than doing something like: iptables - A INPUT -i eth0

personally I would put a hashlimit source ip rule WAAAAY earlier in iptables for better DoS filtering, like:
iptables -t raw -A PREROUTING -i eth0

connlimt requires conntrack so can't be as early as raw. So replace "raw" with "mangle" for your connlimit rule later in iptables. Unlike FORWARD or INPUT, this all still before any Docker or ufw rules and whatnot for incoming packets if you use that too as well, so Docker won't allow them to bypass it :)

Also if you're using ipv6 don't forget to make ip6tables rules too.

Finally before you ask me how many packets you should allow this actually depends on your application. You'll want to tweak either the hashlimit or hitcount limit to what's needed for your application. So act as a client yourself and test the limits of your rules until you find suitable levels of traffic from ""legit" clients is basically what you're doing. This may involve using a packet capture tool such as tshark/wireshark or tcpdump. Then test connlimit next using a tool like nmap to open multiple connections.

That was a VERY thorough answer. Thank you so much! I'll look into the solutions you suggested!

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct