SMTP and HTTP DDOS from spoofed IP addresses

@jadeweb:

Second, and more importantly, is there anything I can do to negate their efforts locally? (Assuming I'm correct about the IP spoofing, all my reading on the topic suggests I don't have any options).
Probably my best suggestion is to try to make throwing away the bad connections as efficient as possible. To be honest, hundreds of connections a minute (just a few a second) should not really be that heavy a load to handle if all you're doing is rejecting them.

You mention php running into memory errors - that may imply that your current apache configuration permits too many apache processes at peak. Assuming bad requests can be rejected fairly quickly, they really shouldn't interfere much - aside from potentially adding a little latency - with real php processing. So one possibility is that you ended up thrashing memory with too many apache workers which exacerbated the impact of the bad connections. Generally you want to make sure that at peak, your working set remains completely within the available memory of your Linode - there are numerous threads in the forum regarding apache tuning.

If that's tuned reasonably, then another possibility is that you're allowing the bad requests to make it through to your php scripts where they require significant processing to reject. If that's the case (e.g,. a random post to / on your server ends up with full php processing) I'd definitely see if you could lock that down within the web server, or exit out at the php layer as quickly as possible. What restrictions you might put in place to implement that likely depends on the application, but it could be as simple as a cookie check (e.g., it's probably unlikely posts are permitted from random anonymous requests).

Assuming that the bad requests can be rejected at the web server level, then there's still the question of how to do that most efficiently. If you are using mod_php (rather than, for example, php-fpm), that also implies that each apache worker is using more memory than needed, since each includes a php interpreter. So, for example, you could front your apache with a server like nginx which could handle rejecting bad requests (or even servicing requests for static resources) more efficiently - only delegating to apache proper requests that need to use it. Or using php-fpm can accomplish something similar since you separate management of php resources from web server resources, so for example the apache workers themselves become less resource intensive.

You might also want to make sure that your default web processing does not serve your main site, but only does so with the proper server name. Won't help if they're targeting your server by its proper name, but in many cases random attempts won't actually know the official web server name for the address they are targeting, so you can reject them sooner. In nginx, I use a "return 444" block as my default host, which doesn't even bother with niceties, but just abruptly closes the socket.

– David

See this thread: https://forum.linode.com/viewtopic.php?f=19&t=10501

He started getting hit with the same type of attack around the same time you did.

You know now I come to think of it I've seen this kind of activity too (only on port 80 I don't have 25 open). What reminded me was the user agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)". I've not seen it to a scale where it'd cause any performance issues but I've seen a lot of 404 requests where that UA has been scanning for phpmyadmin etc. A quick google suggests that UA isn't legitimate, and if it is it'd be IE6 which no sane person should use any more. So you could block the UA at the Apache level so php wouldn't have to process the request.

@db3l — Your second guess is correct; These requests are getting through to PHP (we're using Apache with a CGI wrapper), and in this specific case I can filter out the bad requests easily at the Apache config level since the homepage for this vhost doesn't have a legitimate reason to accept POST requests (That's actually what I was just sitting down to do. :) ) But, this is a multi-tenant server so that approach won't work everywhere (should work great for this specific attack though).

Great suggestion to return 444 on the default host (we just return a lean, branded static page now), but for various reasons it's not practical to use Nginx on this server (I use it elsewhere and love it). I may return a 403 though, as that sounds a bit better than sending out even a minimal static page.

@Main Street James — Excellent find. I hadn't come across that in my research, and there's loads of good info there too. Thank you.

@obs — Good tidbit, and I just did some digging myself based on your notes. I agree on the sane user note, but with >150 sites hosted on this server, I'm hesitantto take that blanket approach until performance is truely impacted.

Thanks again to each of you for the quick and thoughtful replies. Some really helpful comments.

I'm still curious what the purpose of this attack is (They're not actually attempting any exploits, just acting as an annoyance), but it sounds like one of those "who the hell knows" scenarios, so I'll just have to do what I can and be satisfied with bewilderment. :)

@jadeweb:

Great suggestion to return 444 on the default host (we just return a lean, branded static page now), but for various reasons it's not practical to use Nginx on this server (I use it elsewhere and love it). I may return a 403 though, as that sounds a bit better than sending out even a minimal static page.
Note that the "return 444" is a nginx-ism. It doesn't actually return an HTTP code 444, but just shuts down the connection at the socket layer immediately without sending any data at all to the client. I don't think Apache has something equivalent, but certainly returning any reasonable HTTP error code would be roughly equivalent.

– David

@jadeweb:

So, it seems an attacker is spoofing IP addresses to flood SMTP and HTTP ports. To begin with they directed enough traffic to take down services, but without any action on my part, reduced the magnitude of their attack to a point that, though it persists, it is not having a significant impact on resources.

Just FYI, TCP-based protocols (which include both SMTP and HTTP) can't be IP-spoofed like UDP protocols can. This is due to the three-way handshake required to start up a TCP socket. These requests are making it all the way to apache (layer 7), and as such, have to be real fully-connected sockets.

So rather than a spoofed IP, these requests are likely coming from some sort of botnet.