Quite a few 408 errors in Apache log - DoS ?

All those requests are timing out. Could be something like slowloris. What's your MaxClients number?

Those tend to be from web browsers doing predictive optimization; if the browser thinks there's a good chance the user is going to do something, it will prepare to do it. If you're the first result in someone's Google search, or their mouse cursor hovers over a link to your site a little too long, it may open a connection or start loading the page in anticipation of a click.

Sometimes, it guesses wrong.

MaxClients 24

Maybe I should bump that up?

Linode is running a few blogs, indexed in google. Perhaps it is just the mouse over previews? That would make the most sense.

MaxClients 24 looks OK for Linode 1024. I was just wondering if you had it very low, like 5. If your pages load quickly and the load is low, 24 is just fine.

AFAIK, mouseover preview images are served from Google's own servers. It's more likely to be browser pre-loading as hoopycat said, although in that case I'd expect the browser to pre-load at least one page instead of letting the connection time out. Also, it's a bit suspicious that some IPs open 4-5 connections at the same time and let them all time out without sending any requests. But different browsers use different kinds of speed-enhancing tricks, so I might be wrong. I miss the time when browsers just did what I told it to do and nothing else.

You might be able to find out more about those users by searching for some of those IP addresses in your access log. At least a few of them might have actually loaded a page. If so, the log will also contain the browser name. On the other hand, if none of them loaded any actual page, it might be a robot with Slowloris-like behavior.

Ok thanks for your help.

I installed mod_antiloris as a band-aid and will see how it goes over the next few days.

sudo apt-get install libapache2-mod-antiloris

Follow up:

No conspiracy, blogs I run are just getting more traffic. :)

Changed

KeepAliveTimeout 2

to

KeepAliveTimeout 5

Resolved,

@reaktor:

Follow up:

No conspiracy, blogs I run are just getting more traffic. :)

Changed

KeepAliveTimeout 2

to

KeepAliveTimeout 5

Resolved,

All that does is give a user up to 5 seconds to request another item (page, gif/jpg, css, etc) on the same connection instead of 2. Unless you're dealing with a lot of users on slow connections, I'm not sure how this would have helped.

^Bah, you are right. Spoke too soon. 408s are still occurring…

$ tail -f /var/log/apache2/access.log
124.169.122.xx - - [26/Nov/2011:16:52:34 -0800] "-" 408 0 "-" "-"
124.169.122.xx - - [26/Nov/2011:16:52:34 -0800] "-" 408 0 "-" "-"
124.169.122.xx - - [26/Nov/2011:16:52:35 -0800] "-" 408 0 "-" "-"

Hi guys,

We started to have a bunch of 408's on our Apache too.

It started around 10 or 11th of November, shortly after an apt upgrade to 2.2.14.

It's a Ubuntu 10.04 LTS.

I remember googling a bit on what was new on 2.2.14 and newer versions, and some changed to how 408's were logged popup out.

I just don't understand the true nature of it: why for some requests to the same resource it occurs, and others it doesn't?

I could image some being bounces, but even on my tests it happens on a prolong "visit". It seems a bit random, but I have to admit I'm not well educated on this matter at all.

Any thoughts?

Thanks

I'm seeing these 408 errors a bunch on two of my VMs.

Looking closer, I'm seeing these almost exclusively with Chrome and IE9.

I think hoopycat's answer is likely the right one. Both IE9 and Chrome use predictive browsing.

I did see a single one of these 408s for Firefox 8. But I suspect an add-on might have been in use on that one, or maybe it was an actual glitch on the client.

For my part, I'm disregarding them. They're annoying to see in the Logwatch emails, but they don't seem to be a real threat.

@hybinet:

AFAIK, mouseover preview images are served from Google's own servers.

I believe this to be true as well; iirc, when google creates said previews, it also uses a UA of "Google Web Preview" (not exact, just contains this).

Unfortunately, the IP blocks that Google uses for this service are shared by Google App Engine, which I've been hearing from many server admins, has had a lot of abuse pouring out from it lately.

Another data point…

I've been experiencing the same thing and it's been driving me nuts because I haven't been able to figure out what's causing it! At first, I feared it had something to do with my company's Apache/PHP/MySQL/Javascript/Node.js/Socket.IO web app (also on Ubuntu) because I saw a lot of 408's coming from our customer's IPs. But then, I found one entry in the server logs where someone got a 408 browsing the website for our web app, as opposed to the web app itself:

82.194.219.xxx - - [20/Jan/2012:15:28:09 -0800] "GET / HTTP/1.1" 200 2752 "http://www.google.no/url?sa=t&rct=j&q=rundown%20program%20for%20tv%20production&source=web&cd=2&ved=0CCoQFjAB&url=http%3A%2F%2Fwww.rundowncreator.com%2F&ei=gfgZT_WoHIvY8QPP1eCVCw&usg=AFQjCNHY0cBhQcY-bvO6pfadAnOOhWDDYA" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Input.css?r=7 HTTP/1.1" 200 849 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /GoogleAnalytics.js?r=7 HTTP/1.1" 200 686 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Functions.js?r=7 HTTP/1.1" 200 470 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Style.css?r=7 HTTP/1.1" 200 1878 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /index.js?r=7 HTTP/1.1" 200 838 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /jQuery.js?r=7 HTTP/1.1" 200 33647 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Rundown-Creator-logo-385-to-343x65.png HTTP/1.1" 200 4519 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /More-features-button-207x41.png HTTP/1.1" 200 7853 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Plans-and-pricing-button-207x41.png HTTP/1.1" 200 7476 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Screenshots/Slideshow-teleprompter-960x513.jpg HTTP/1.1" 200 74504 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Screenshots/Slideshow-rundown-960x513.jpg HTTP/1.1" 200 78612 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Screenshots/Slideshow-script-editor-960x513.jpg HTTP/1.1" 200 81354 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:10 -0800] "GET /Used-by-371x45.png HTTP/1.1" 200 9037 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:11 -0800] "GET /Favicon-Transparent-48x48.png HTTP/1.1" 200 1168 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /features/ HTTP/1.1" 200 3561 "http://www.rundowncreator.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/On-air-show-timer-446x327.jpg HTTP/1.1" 200 41148 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/User-management-622x470.jpg HTTP/1.1" 200 37132 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/TV-radio-rundown-software-622x470.jpg HTTP/1.1" 200 66498 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/Teleprompter-622x470.jpg HTTP/1.1" 200 51941 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/Chat-622x347.jpg HTTP/1.1" 200 54788 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:28 -0800] "GET /Screenshots/TV-radio-script-editor-522x541.jpg HTTP/1.1" 200 49598 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:29 -0800] "GET /Screenshots/TV-radio-rundown-software-on-iPad-and-iPhone-359x333.jpg HTTP/1.1" 200 24355 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:28:29 -0800] "GET /Browsers-500x128.jpg HTTP/1.1" 200 26525 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:29:27 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:29:27 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:29:27 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:29:27 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:29:28 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:29:28 -0800] "-" 408 0 "-" "-"
82.194.219.xxx - - [20/Jan/2012:15:30:08 -0800] "GET /about-us/ HTTP/1.1" 200 3009 "http://www.google.no/url?sa=t&rct=j&q=rundown%20program%20for%20tv%20production%20free&source=web&cd=5&sqi=2&ved=0CEQQFjAE&url=http%3A%2F%2Fwww.rundowncreator.com%2Fabout-us%2F&ei=5vgZT9e6GsfU8QOVpJiZCw&usg=AFQjCNG8iDbpTZMbXsD4xB0jByKqevxS2A" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:30:08 -0800] "GET /Jeff-360x233.jpg HTTP/1.1" 200 48253 "http://www.rundowncreator.com/about-us/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:32:05 -0800] "GET /about-us/ HTTP/1.1" 200 3009 "http://www.google.no/url?sa=t&rct=j&q=rundown%20program%20for%20tv%20production%20free&source=web&cd=5&sqi=2&ved=0CEQQFjAE&url=http%3A%2F%2Fwww.rundowncreator.com%2Fabout-us%2F&ei=5vgZT9e6GsfU8QOVpJiZCw&usg=AFQjCNG8iDbpTZMbXsD4xB0jByKqevxS2A" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:32:57 -0800] "GET /features/ HTTP/1.1" 200 3561 "http://www.rundowncreator.com/about-us/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:34:56 -0800] "GET /plans-and-pricing/ HTTP/1.1" 200 3225 "http://www.rundowncreator.com/features/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
82.194.219.xxx - - [20/Jan/2012:15:34:56 -0800] "GET /PlansandPricing.js?r=7 HTTP/1.1" 200 581 "http://www.rundowncreator.com/plans-and-pricing/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"

Our website is pretty straightforward. There's nothing really crazy going on. So that leds me to believe that hoopycat must be right and it's a predictive browsing thing. Whew.

The most frustrating thing about these 408's is that there's no URL, no user-agent, no referer, nothing except the IP address. This is not surprising, since technically these aren't HTTP requests at all.

Perhaps Apache shouldn't even log them in the access log. The access log is for logging HTTP requests, and these aren't HTTP requests, just idle TCP connections. Browsers probably don't expect to get a 408 response from an idle connection, either.

An entry in the error log along the lines of "Client x.x.x.x closed connection without sending any requests" would be more informative. After all, the error log is where other connection-related entries such as "MaxClient reached" gets logged.

> Anyone ever seen this with apache? Or is this just a 'meh'.

I'd say it's just a meh. The 408 error you mention is just a simple request timeout; a passive attack at most.

Anyone who's been through the logs of a highly trafficked website has seen hell. I get a couple hundred (on a good day, bad day is thousands) hack attempts which end up in my access logs. I've seen random characters being appended to perfectly valid URLs making them into 404s. I've seen a remote client requesting the same URL 3000 times even though it gives a "403 Forbidden Error" every single time.

I just love the internet :twisted:

I've also been seeing a lot of 408's especially this past couple of weeks, most from .ru (Russia) domains.

I get tons of those as well. All coming from IE9 and exactly 14 seconds after another request. I couldn't replicate it with my own IE9 though :-/

It's a bit random if it occur after a script or not, except for one ping-script I have, which doesn't use KeepAlive (header "Connection: Close").

This ping-script is called from a Flash file, so it can't be google-mouseover-previews.

The error-log is filled with: "Request header read timeout"

I'm planning upgrade Apache to 2.4 an see if it changes anything.

I'll keep you posted.