Does this mean I'm getting DDos'D?
> tail -f -n 0 error.log
[Sun Jan 09 19:34:48 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:49 2011] [error] [client 88.90.106.161] Invalid URI in request \xf2\xa8\xe4\v5E\x19\x84\xc0\x04\xff\xaa
[Sun Jan 09 19:34:50 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:51 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:51 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] Invalid URI in request #\xebM&\x0c\xb8\x8f\xa9\xe9-Ht\xe8S\xae\xf4\x90"-\xf7I\xf2
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] Invalid URI in request Z\xff\x81Y\xacR\xee\x1eu\xed}\x94\xeb=\xb5\xb3z\x10\xfe\xd3g\xa7dc\xb3\x0c\x98\xddB\xa67\xf9<
[Sun Jan 09 19:34:53 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:53 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] Invalid URI in request \xc3x \x01>\x9a@M\x97\x17r
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:55 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:56 2011] [error] [client 88.90.106.161] Invalid URI in request \xf2\xa8\xe4\v5E\x19\x84\xc0\x04\xff\xaa
[Sun Jan 09 19:34:56 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:57 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:57 2011] [error] [client 88.90.106.161] Invalid URI in request #\xebM&\x0c\xb8\x8f\xa9\xe9-Ht\xe8S\xae\xf4\x90"-\xf7I\xf2
[Sun Jan 09 19:34:58 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:35:00 2011] [error] [client 88.90.106.161] request failed: error reading the headers
Happens really fast and brings down the server to a halt. Many apache instances and I have to restart apache for it to work. It'll stop working within 3 seconds.
I've blocked the IP for now. Something similar happened 3 days ago from a similar ip.
Both seem to be from same provider (norway).
I sent email to abuse-mailbox:
9 Replies
> tail -f -n 0 error.log
[Mon Jan 10 17:29:14 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:16 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:17 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:18 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:19 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:20 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:22 2011] [error] [client 83.109.195.192] request failed: error reading the headers
Is there anything I can do to prevent it from taking down Apache? Some type of measure to throttle a specific ip if it's requesting too much in a short amount of time? Should I tweak my Apache settings? This is a problem if I'm not around the current second.
Should I setup monin to restart apache automatically? Anyone have some tips/solutions so this doesn't take down the server for a hour+ until I get back to computer?
So in summary, server seems to be getting hit same time every night from a different IP but all related to this "norway isp" telenor.
If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.
– David
@db3l:
Even if you get hit with a ton of these it really shouldn't take down Apache. At worst it should just make your site slow while they are happening, but it certainly shouldn't interfere with basic server function or access (e.g., over ssh).
If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.
– David
David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.
This is my config:
>
MinSpareServers 3
MaxSpareServers 6
ServerLimit 12
MaxClients 40
MaxRequestsPerChild 3000
I'm on 756 MB of memory.
I would change the MaxClients to 20 but I don't want to change it arbitrarily without testing it to see if it makes a difference.
I tried to use a "ab" benchmark to see if it would bring the server down but no luck. The server kept responding promptly. Should I leave it on longer? Any ideas?
What would be a good ab benchmark test?
I tried doing the following:
ab -n 5000 -c 10
Which actually ended up closing my local wifi/internet connection. I originally tried -n 1000 -c 5. No luck on slowing down server.
Any advice would be greatly appreciated! Thank you!
@arachn1d:
David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.
Well, it sounds like you have an almost nightly occurrence via external sources, so even if you can't generate the necessary load manually, tweak and then just wait a day :-)
Though as hybinet points out, the crucial point is how many simultaneous requests you initiate, not really the total number, since that's what causes all the Apache clients to be instantiated, and likely overloading your memory. You ought to be able to generate quite a few simultaneous connection attempts even over a slow client link.
It depends on your application stack, but MaxClients of 40 feels too high for a 768. Personally I'd have no worries about dropping it to 20 and then watching behavior. Odds are your normal usage won't be affected at all.
There are a number of Apache tuning threads here in the forums you might reference for other ways to benchmark your system. One at
BTW, I had similar log patterns on an Apache server of minte in the past, and I've been wracking my brain trying to remember what I identified it as, as it didn't turn out to be malicious. Finally remembered…
In my case it ended up being incoming BitTorrent traffic, where a client behind the same firewall as the server had ended up choosing the same (non-standard) port as was being forwarded to Apache. So when the client stopped, for a period of time thereafter, I got incoming probes from BitTorrent clients looking for my peer, but now landing on Apache.
In general, your logs just say that some client using a binary protocol is connecting to the port Apache is listening on. So your case could be completely different, or could in fact be malicious, but perhaps this will help you think of other possibilities, especially if there's any chance that Apache is listening on a port that may be getting other uses, or proxy's or whatever.
-- David
In that case, you could set up nginx on port 80 and have it proxy all requests to Apache (maybe except some static files, which are better served by nginx). Ideally, nginx will see that the request is not valid HTTP, and drop the connection without passing it on to Apache. Also, a few hundred additional connections on nginx will hardly use any resources, while a few dozen additional connections on Apache can easily bring a VPS to its knees.