View All Products
Compute
Storage
Networking
Databases
Services
Developer Tools
Industries
Pricing
Community
Engage With Us
Community ▶ Questions ▶ Help! DDoS attack, at a complete loss...
Log in to Ask a Question

Help! DDoS attack, at a complete loss...

general

We've been under DDoS attack since Friday. From Friday until around 10 hours ago, I thought it was something else, then ran a netstat command:

netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

Which returns almost 200 IPs trying to connect to the server (http://pastie.org/2410271), the top ones being:

9 86.111.75.8

10 195.245.97.181

10 95.30.133.238

11 178.125.237.157

14 31.43.82.236

14 95.135.19.200

15 178.124.84.155

16 92.112.165.39

18 78.30.228.147

112 2.134.153.23

All of these IPs are in random foreign countries. We get about 30,000 unique visitors per month, 99% in the US, so this must be a DDoS attack.

I have exhausted seemingly all options and I do not know what to do.

-Installed modsecurity and modevasive.

-Added a rule to IPTABLE.

-Binded mysql to 127.0.0.1

This is my config for mod_evasive:

DOSHashTableSize 3097

DOSPageCount 2

DOSSiteCount 20

DOSPageInterval 1

DOSSiteInterval 1

DOSBlockingPeriod 10

DOSLogDir /var/log/apache2/mod_evasive

DOSEmailNotify alex@rentabilities.com

DOSWhitelist 127.0.0.1

I even created a new linode, and changed our DNS. Right away, the attack started happening on the new linode. So that leads me to believe they're attacking our hostname.

I can't think of any reason anyone would want to attack us. Our hostname is rentabilities.com - we're a startup that makes it easy to rent anything (ala ebay for renting), and as far as I know we don't have any enemies :-)

Has anyone experienced DDoS attacks before and found a way to solve it?

We're on a 4GB linode. Maybe resizing would help, but I somehow doubt it.

Any help is truly appreciated!

17 Replies

Have you taken a look at your Apache logs to see if the IPs are trying to access anything in particular? Maybe that would give a clue as to why they're attacking you.

@Sophira:

Have you taken a look at your Apache logs to see if the IPs are trying to access anything in particular? Maybe that would give a clue as to why they're attacking you.

Thanks for the reply. Good idea. Sample outputs:

2.134.153.23 - - [22/Aug/2011:01:46:37 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:39 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:39 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:42 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

I don't see specific files being requested - will have to dig more in the AM.

Your site is currently unavailable to me here in the uk - not sure if that is intentional or not.

I raised this very issue a couple of days ago. I am a small business too and just woke up to this problem.

Please let us know how you get on. Have you had a demand for money or anything?

How this is able to go on is disgraceful and the dirty secret of doing business on the internet. It is run by the mafia's not the service providers is the truth.

Down here ( Sri Lanka ) too !

Thanks for checking. The site's meant to be accessible anywhere, and we're still under dos attack.

Do all the requests use that same Russian Firefox user agent string, or is there some other consistent discriminant? If so, you could put nginx as a proxy in front of Apache, and use it to drop the connections for the bogus requests before they make it to Apache. Google nginx 444.

Configured properly, nginx should be able to handle this type of load without breaking a sweat, especially if it just has to drop the connections as they come in.

What effect is it having on the Linode as a whole, and is your network traffic graph showing anything interesting? Is anything showing up in Apache's error log, specifically about MaxClients?

I'm suspecting it's "just" a whole bunch of connections eating up your available connections, such as what Slowloris would do against Apache+mpm-prefork. Still a bugger, but an important distinction for mitigation strategies.

Is Cloudflare helping things at all?

Thanks. I think the requests are coming from several bots (not just the Russian one):

220.255.2.171 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 0 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

95.57.90.155 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

31.162.226.245 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

178.123.54.251 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

178.125.44.203 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

79.124.194.249 - - [22/Aug/2011:12:11:05 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

79.124.194.249 - - [22/Aug/2011:12:11:05 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

2.54.13.48 - - [22/Aug/2011:12:11:06 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

78.30.228.147 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

178.178.42.124 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

223.207.129.136 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 13600 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

79.124.194.249 - - [22/Aug/2011:12:11:33 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

95.30.133.238 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

81.90.238.41 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

93.116.90.49 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

178.90.25.164 - - [22/Aug/2011:12:11:47 -0400] "GET / HTTP/1.1" 200 27360 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

178.123.54.251 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33261 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

92.133.73.130 - - [22/Aug/2011:12:12:07 -0400] "GET / HTTP/1.1" 200 21900 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

178.89.47.254 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"

I'll check that out!

What's in the mod_evasive log? I'm surprised it's not catching those repetitive GET / requests.

Thanks - you're right.

I did some more digging and found that there wasn't any log entries being created for mod_evasive. So I competely uninstalled it, then re-installed it using this article as a guide:

http://www.linuxlog.org/?p=135

That seems to have at least got it logging and working. In the log I now see:

-rw-r–r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

I'd expect this to grow to hundreds of connections in the next few minutes after starting Apache again.

Looks like that log is changing.

-rw-r–r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:01 dos-173.9.107.126

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:00 dos-77.124.17.60

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

… and growing:

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-109.251.20.66

-rw-r--r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-118.8.26.167

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:01 dos-173.9.107.126

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.125.100.101

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-178.125.142.124

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-178.125.68.231

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-178.137.4.41

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.216.122.233

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.90.192.242

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-180.1.225.47

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-180.31.144.15

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-188.237.65.176

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-188.99.187.65

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-189.239.209.80

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-190.226.214.165

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-190.41.236.215

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-193.151.105.5

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-193.19.228.226

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-193.84.22.10

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-194.44.235.187

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-200.35.36.75

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-210.117.21.95

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-212.142.93.141

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-46.10.64.96

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-46.118.189.216

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-46.162.233.223

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-62.221.107.177

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-62.87.214.17

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-74.7.76.233

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-76.251.218.25

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-76.90.202.228

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-77.101.245.155

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:00 dos-77.124.17.60

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-77.124.67.27

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-77.196.114.94

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-78.94.204.151

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-79.39.116.172

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:03 dos-81.90.238.41

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-82.104.2.117

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-82.204.95.254

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-82.82.177.184

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-83.251.60.28

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-83.99.181.74

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-84.101.142.64

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-84.229.255.32

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-84.3.52.55

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.34.25

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.63.202

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.75.89

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-85.114.11.98

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-85.64.157.98

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-87.20.222.21

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-87.91.125.225

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-91.156.164.2

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-91.22.137.172

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-92.119.57.241

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-94.76.66.32

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-95.170.191.6

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238

-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-95.57.70.25

-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

mod_evasive turned out to be evasive, but not evasive enough.

A friend stepped in and helped me make two changes

1. Blocked common user agents.

We added these 4 lines to .htaccess:

BrowserMatch "ru" ddos

BrowserMatch "1.1.4322" ddos

Order Deny,Allow

Deny from env=ddos

2. Setup CloudFlare

http://cloudflare.com

In the Apache config, you have to add CloudFlare's IPs to the whitelist, for example:

DOSWhitelist 127.0.0.1

DOSWhitelist 204.93.240.0

DOSWhitelist 204.93.177.0

DOSWhitelist 199.27.128.0

DOSWhitelist 173.245.48.0

DOSWhitelist 103.22.200.0

DOSWhitelist 141.101.64.0

Will post back here as I progress… hopefully this helps someone else out in the future.

Looks like your site is back up and running :)

I'm having trouble with it, though. I searched for A/V equipment near Montreal, and got nothing. Then I clicked on the A/V section and saw a bunch of stuff. A projector listed for $249. I clicked on it, and it shows the projector is actually $429, and is "Available from 1 store near Montreal" with the store being in Melbourne, Australia. Which, being 17 thousand kilometers away on a planet where the maximum distance between any two locations is a bit over 20 thousand kilometers, isn't exactly down the street ;)

I'm really glad you got it sorted :)

I would consider rejecting "ru" as standard except I would be afraid it might get me some unwelcome attention. I was spending the day looking at ways to mitigate just such an attack when your post popped up on the boards.

I would not suggest to block 'ru' or any other particular languages 'cn' for example. You can do it temporary to stop the attack for the moment, but unblock it after attack is gone.

Nginx, CloudFire probably the good choice. You also have more expensive solutions from VeriSign for example.

Also I would suggest to have more then one WEB servers, and one or two DB servers. You will need to configure load balancer for it.

It will not stop the attack, but it will help you to survive it.

Since the last post to this thread was three months ago., I think it's safe to say that help isn't needed any more.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Log in to Reply
Community Code of Conduct