DNS DDoS Attack - Resolved

If you really wanted a new IP address you could create a new Linode, copy your disk image over from your current Linode then delete your current Linode.

Edit: Not that that's really what you should do to fix this…

@GLaDOSDan:

If you really wanted a new IP address you could create a new Linode, copy your disk image over from your current Linode then delete your current Linode.

Thanks, but I've paid in advance for this linode so that isn't an option. Or am I mistaken about that?

Edit: Yes exactly :) Right now I've moved all of my vhosts to another linode and am just monitoring. The thing that sucks is in my opinion the linode is unusable…

Just wanted to let you guys know that linode changed my IP (thank you linode!). I'm up and running with no more DNS noise.

Damn this attack is a total PITA.

@asp:

@GLaDOSDan:

If you really wanted a new IP address you could create a new Linode, copy your disk image over from your current Linode then delete your current Linode.

Thanks, but I've paid in advance for this linode so that isn't an option. Or am I mistaken about that?

Edit: Yes exactly :) Right now I've moved all of my vhosts to another linode and am just monitoring. The thing that sucks is in my opinion the linode is unusable…

If you paid in advanced and make a new box you can ask linode to remove the remaining time over to the new one.

I'm curious, why did this attack make the node "unusable"?

It's a 1.5Mbps attack, if you're dropping the traffic it would have absolutely no impact on your Linode except to accrue roughly $48/mth worth of bandwidth usage, which isn't terribly much, all things considered.

@Guspaz:

It's a 1.5Mbps attack, if you're dropping the traffic it would have absolutely no impact on your Linode except to accrue roughly $48/mth worth of bandwidth usage, which isn't terribly much, all things considered.
Inbound transfer is free. If you're dropping it, it costs you nothing. Heck, it probably even improves Linode's ratios.

Good point :)

So, yeah, I'll revise my statement to "A 1.5 Mbps attack that is being dropped by your firewall should have no impact whatsoever on your linode" :)

@mnordhoff:

I'm curious, why did this attack make the node "unusable"?

At the time of my writing it was 1.5Mb/s but it was as high as 7Mb/s. Yes I was dropping it, but that was also eating up CPU cycles of which I saw as much at 15%. All of that is right off of the top of the linode (L768).

I didn't want to continue using the linode when there was an active attack against it so that made it unusable to me. I was totally patient though as I had moved critical sites over to another linode and was hoping it was just stop, but that didn't happen.

Would you mind explaining your comment a bit more (or point me in the right direction) about how it would improve Linode's ratios?

(Sorry for the delayed respond…)

And thanks a lot for the comments I appreciate it! :)

@asp:

At the time of my writing it was 1.5Mb/s but it was as high as 7Mb/s. Yes I was dropping it, but that was also eating up CPU cycles of which I saw as much at 15%. All of that is right off of the top of the linode (L768).
OK, but 15% of 1 core is nothing. Even 15% of your overall CPU power of 4 cores – i.e. 60% of 1 core -- shouldn't cause problems. (Well, at 60% I'd start worrying about the networking stack a bit, especially if the packets go through conntrack.)

@asp:

Would you mind explaining your comment a bit more (or point me in the right direction) about how it would improve Linode's ratios?
It was largely a joke. Settlement-free peering agreements between ISPs – when they connect each others' networks for no money -- often place great importance on their traffic ratios, requiring that they exchange a relatively equal amount of traffic. Linode is probably pretty unequal, since a lot of their traffic is probably web stuff, which tends to use more outbound traffic. (HTTP request: 1-2 KB. Response: Anything, but frequently hundreds of KB.)

@mnordhoff:

OK, but 15% of 1 core is nothing. Even 15% of your overall CPU power of 4 cores – i.e. 60% of 1 core -- shouldn't cause problems. (Well, at 60% I'd start worrying about the networking stack a bit, especially if the packets go through conntrack.)

In this case I was dropping these packets and not tracking them. I guess I'm (incorrectly?) hung up on the fact that regardless what the numbers are legitimate traffic would be competing with this DNS noise and that just doesn't sit right with me. I suppose the counter argument to that is there is plenty of network noise, but it doesn't come in the form of several thousand packets a second :).

So would you have just written this one off? Now you got me thinking that I was over reacting, but I do want to make sure I have a realistic view in case this happens again. I was really lucky to have space elsewhere to move sites around but that won't always be the case.

What other approaches would you have taken to mitigate the attack if any? Like I said I contacted abuse departments and whatnot (but that is really a waste of time bc the packets were likely forged), but other than that there didn't seem to me much else I could do.

@mnordhoff:

It was largely a joke. Settlement-free peering agreements between ISPs – when they connect each others' networks for no money -- often place great importance on their traffic ratios, requiring that they exchange a relatively equal amount of traffic. Linode is probably pretty unequal, since a lot of their traffic is probably web stuff, which tends to use more outbound traffic. (HTTP request: 1-2 KB. Response: Anything, but frequently hundreds of KB.)

Thanks for the explanation!

There's a difference between "don't want to use because the packets don't sit right with me" and "the linode is unusable". Let's be clear, the linode is perfectly usable (for anything but DNS serving). If I said "I don't want to use my linode because it's a full moon and also a Tuesday", that doesn't somehow make my linode unusable (unless it's a werelinode, but that's another issue). It just means that I haven't restocked on silver USB keys recently.

Basically, you don't pay for inbound traffic, so if you just drop the inbound DNS traffic, there is no impact on your linode. You've got four cores to play with (so effectively 400%), so 15% usage isn't a problem unless you're maxing out all four cores. The only scenario where you might see some impact is if you're trying to run a DNS server, but there's not really any reason why you would since Linode provides free DNS servers both for resolution and hosting.

Edit: apologies for the vagueness of my "unusable" statement please let me clarify below…

@Guspaz:

There's a difference between "don't want to use because the packets don't sit right with me" and "the linode is unusable". Let's be clear, the linode is perfectly usable (for anything but DNS serving). If I said "I don't want to use my linode because it's a full moon and also a Tuesday", that doesn't somehow make my linode unusable (unless it's a werelinode, but that's another issue). It just means that I haven't restocked on silver USB keys recently.

What I said was @asp:

I guess I'm (incorrectly?) hung up on the fact that regardless what the numbers are legitimate traffic would be competing with this DNS noise and that just doesn't sit right with me.

In other words legitimate traffic would be competing with the traffic from the attack and that's what didn't sit right with me. Does the linode work? Well yes I said that in my inital post, but I followed that I'm not going to use a machine for business purposes while it's under attack thereby making it unusable to me.

I really don't think the previous statement is unreasonable. I apologize if I made it sound like linode's service is not totally awesome because it is. I've been a very happy linode customer for years, but this has never happened to me before which is why I was asking the community for help/perspective.

So you would have just ignored the attack, or noticed it an just said "whatever…" based on your statements below?

@Guspaz:

Basically, you don't pay for inbound traffic, so if you just drop the inbound DNS traffic, there is no impact on your linode. You've got four cores to play with (so effectively 400%), so 15% usage isn't a problem unless you're maxing out all four cores. The only scenario where you might see some impact is if you're trying to run a DNS server, but there's not really any reason why you would since Linode provides free DNS servers both for resolution and hosting.

BTW it's freaking awesome that we don't have to pay for inbound traffic. I really would have been screwed otherwise. I was also saved by the fact that I was alerted when the inbound connections got to a certain rate…

Thanks again for the help!

@asp:

but I followed that I'm not going to use a machine for business purposes while it's under attack thereby making it unusable to me.
You might want to define "attack" a bit better.

There are attacks, and then there are ATTACKS.

Eating up a bit of inbound pipe and a few clock cycles isn't really worth getting your cyberpanties in a bunch over.

If people stopped using systems for every little attack, a few port scans would shut down the internet.

You did what you thought best, but probably need to learn how to mitigate such things in the future instead of packing up shop and moving across the street the first time your shop wall gets tagged with a bit of graffiti.

@vonskippy:

You did what you thought best, but probably need to learn how to mitigate such things in the future instead of packing up shop and moving across the street the first time your shop wall gets tagged with a bit of graffiti.

Vonskippy that IS exactly what I was asking here! About how others have mitigated such attacks and in general thoughts on this type of attack with respect to their linode.

The sort of message that I'm getting from a few people that have commented so far is: "I'm a stupid idiot who should just ignore the DNS noise because A) It doesn't cost any money, and B) It doesn't really affect the linode (other than not allowing me to run a DNS server and burning up CPU cycles)."

But no one has actually said that so I don't want to assume. So again I'll ask how would you approach this? Would you just ignore the traffic and carry on?

As I stated previously I had a firewall up and running and was dropping all of the traffic. It just so happened that I was able to move things around but I can't always count on that (as you have suggested). I never said that was the best approach, but I needed to take some sort of action so that I could work on the linode and investigate the cause of the problem further.

I've done some research on the problem (prior to posting to the community) which I've found some good information here:

http://seclists.org/nanog/2012/Apr/108

In fact my problem was identical to the poster. I wasn't running a DNS daemon or listening on port 53.

But other than that I really couldn't find anything that talked about how I could mitigate the problem. That's when I thought I'd check in with the community…

I ignore them, happens to me all the time some box or group of boxes for some reason decide to try and poke holes in one or more of my servers, in the past year I'd say only one has actually had any effect and that was on a pretty loaded server so I just dropped anything that looked suspicious, they stopped after a few days.

What's really sad is lately I've noticed an increase in people trying to perform syn flood attacks which is very easy to mitigate.

Well, the definition of mitigating an attack is pretty much reducing the impact of the attack such that regular traffic can be served at a reasonable speed. Getting 1.5 Mbps of traffic on a 1000 Mbps pipe isn't going to impact regular traffic. You say that legitimate traffic would be "competing" with the attack, but you're talking about something that represents 0.15% of the total pipe coming into the box. How is competing with a measly 1.5Mbps different from competing with the inbound traffic of all the other linodes on the physical host? What if somebody else on the host is running a legitimate DNS server getting more than 1.5 Mbps of traffic?

If you're under attack, and the attack is having zero impact on either your financials or ability/performance to serve traffic, I'd call that attack effectively mitigated.

So, in response to your question, if somebody sent 1.5 Mbps of malicious DNS traffic to me and I didn't have anything listening on port 53, yes, I'd just ignore it. Attacks that have no impact are best ignored, because the attacker will eventually give up for lack of effect. But 1.5Mbps barely qualifies as an attack, it's barely more than a probe.

@obs:

I ignore them, happens to me all the time some box or group of boxes for some reason decide to try and poke holes in one or more of my servers, in the past year I'd say only one has actually had any effect and that was on a pretty loaded server so I just dropped anything that looked suspicious, they stopped after a few days.

What's really sad is lately I've noticed an increase in people trying to perform syn flood attacks which is very easy to mitigate.

Thanks obs

Just wanted to thank everyone that replied on this thread. I'm marking it as resolved it seems like the overall consensus is that this really wasn't a big deal at all, be it 7Mb/s or 1.5Mb/s. As you guys have described the overall load placed on the linode and the network pipe is insignificant and ultimately not worth worrying about in this situation.

I'm am glad, however, linode thought it was a reasonable request to change my IP. That made the whole thing a moot point, but I'm grateful for the discussion and all of your ideas.

Thanks again,

Aaron

…well I'd buy you a fur coat (but not a real fur coat, that's cruel)

@derfy:

…well I'd buy you a fur coat (but not a real fur coat, that's cruel)

Eat the whales!

Persons for the

Excessive

Tasting of

Animals

@zunzun:

@derfy:

…well I'd buy you a fur coat (but not a real fur coat, that's cruel)

Eat the whales!

Persons for the

Excessive

Tasting of

Animals

I'd buy you a green dress (but not a real green dress, that's cruel)

I'd buy you furniture for your house (maybe a nice chesterfield or an ottoman)