DDoS attacks and the Three Strikes "Policy"
27 Replies
caker
Linode Staff
@harmone:
So if I get ddosed you cancel my linode subscription?
No, but what I've done in the past is three strikes and you're out (or less) – especially if the severity of the attacks affects all the machines, and subsequently all of our customers in a given data center. There may be other circumstances in which I'd overlook this "rule", but in general Linode cannot afford to harbor the type of clients that attract attacks.
I take the integrity of our network and the quality of our subscribers very seriously. This type of activity on our network won't be tolerated. Clients that attract this type of stuff will need to look elsewhere for hosting.
Thanks,
-Chris
caker
Linode Staff
@dbuckley:
Could you be a bit more specific on the types of things that attract attacks…?
Honestly, I don't know if I can qualify it. My guess is script kiddies getting into juvenile arguments with one another, attacking others from a Linode which prompt a counter attack, that kind of stuff. Who knows.
-Chris
caker
Linode Staff
-Chris
But it would be nice to know that if these things happen often and I manage to develop a popular service, that I wouldn't have to change provider just because some 12 year old child decided to try his botnet on me.
Is this common practice? Do other server providers cancel customers accounts on ddos? Perhaps there are ways to defend oneself against ddos attacks that your isp is willing to share with you.
If a 12 year old controls a 1000-node botnet, wouldn't the traffic generated from one of his nodes look similar enough to each of the other of his nodes, to be possible to block at the router level?
There are some of us who work very hard at running polite and fair IRC servers, but random attacks do happen to any service - just ask Steve Gibson. In any instance I'd whack the power button on my Linode the instant I saw an issue generated from it, I know what it's like to be one of the customers near a DDoS attack and there are some people that start IRC networks on bad pretenses and then provoke people with it.
So thank you for not taking others stupidity out on the rest of us
@caker:
@harmone:So if I get ddosed you cancel my linode subscription?
No, but what I've done in the past is three strikes and you're out (or less) – especially if the severity of the attacks affects all the machines, and subsequently all of our customers in a given data center. There may be other circumstances in which I'd overlook this "rule", but in general Linode cannot afford to harbor the type of clients that attract attacks.I take the integrity of our network and the quality of our subscribers very seriously. This type of activity on our network won't be tolerated. Clients that attract this type of stuff will need to look elsewhere for hosting.
isn't there a middle ground?
if someone is running a legit site and they happen to be a lightning rod one solution would be that they agree to allow their site to be temporarily powered down*. possibly for either x hours or x days–whatever is necessary to wait out the attack.
that way the entire network isn't suffering and owner of the ddos'd site is not being punished (by losing their linode account) because someone else chose to issue a ddos attack against them.
i agree that linode shouldn't be providing ddos protection--it's not in the contract. at the same time, if there are reasonable steps that can be taken to support linode customers who may be the unfortunate target of a violent act, this would reflect positively on linode.
--
*powered down: it seems like this would be a reasonable clause to include in the linode contract. ~"if your account is ddos'd it will be automatically and temporarily disabled to prevent the ddos attack from affecting other customers."
@besonen:
~"if your account is ddos'd it will be automatically and temporarily disabled to prevent the ddos attack from affecting other customers."
Don't DDOS attacks normally involve just throwing as many packets as possible at the target's IP, thereby saturating their network connection? In that case I don't see how powering down the target's linode would prevent other Linode customers from being affected.
If the target becomes unreachable, the attacker would be thinking "LOL my 4tt4ck iz w0rk1ng" not "Darn, they foiled my attack by shutting down their machine, might as well call it off".
@piglet:
Don't DDOS attacks normally involve just throwing as many packets as possible at the target's IP, thereby saturating their network connection? In that case I don't see how powering down the target's linode would prevent other Linode customers from being affected.
You are correct. Disabling the target has no effect on the attack. The affected IP has to be null routed by Linode's network connectivity supplier (Hurricane Electric, in this case). That costs Linode time and money - hence the 'no prisoners' approach to this type of problem.
People who have botnets are the most juvenile people on the planet.
But Linode's policy is a good one. If you want DDoS protection, it will cost you . . . a lot.
@pclissold:
The affected IP has to be null routed by Linode's network connectivity supplier (Hurricane Electric, in this case). That costs Linode time and money - hence the 'no prisoners' approach to this type of problem.
does null routing have to be costly? it could be automated to occur when a ddos attack is detected.
@besonen:
does null routing have to be costly? it could be automated to occur when a ddos attack is detected.
Hurricane Electric would need to spend money to do the detection. From their perspective, one person's DDOS looks like somebody else's busy day. They would have to log traffic for each IP and look for sudden changes. At the moment they only have to log cumulatively at the address block level for traffic billing.
@pclissold:
@besonen:does null routing have to be costly? it could be automated to occur when a ddos attack is detected.
Hurricane Electric would need to spend money to do the detection. From their perspective, one person's DDOS looks like somebody else's busy day. They would have to log traffic for each IP and look for sudden changes. At the moment they only have to log cumulatively at the address block level for traffic billing.
the burden of identifying the ddos'd IP(s) could be placed on Linode. then all that Hurricane Electric (HE) would have to do would be to set the null route.
Linode could create a mechanism for quickly identifying ddos attacks and automatically reporting them to HE for null routing.
Caker, how much work would it be to create a mechanism that automatically id'd a ddos'd IP and reported it to HE? and can you think of any reasons why you wouldn't want to to have such a system in place?
Simple solution: stop doing what is causing you to get DDoS'd. Period.
Unfortunately, taking the high-brow route "But I'm running an anti-spam service that stops spam! It's for the good of the internets!" doesn't warm over spammers. This is just one example. Even if you think you're in the right, you're wronging other customers who share your netblock and UML host here at linode.
Caker's policy is -very nice- and he has to take care of the rest of his customers. I don't know what the individual policies are between the different centers, but Caker & Company may be eating part of, if not all of a LARGE BILL for the bandwidth caused by a DDoS.
@warewolf:
Simple fact of the matter: if you are repeatedly getting DDoS'd, then you are, or have done something wrong. Period.
Simple solution: stop doing what is causing you to get DDoS'd. Period.
you can say that it's a black and white issue, but that doesn't make it so. and it does little to contribute to a thoughtful discussion, imho.
i'm guessing that maybe you are frustrated that someone would even inquire about the nuances of Linode's ddos policy because you imagine that that might mean that said person is in some way unaware of the great service that Caker offers. and that that said person is in some way ungrateful for all that Caker does.
if this is the case, i assure you that someone can be fully aware and appreciative of Caker's efforts and still be interested in engaging in a discussion of how Linode is run.
so, warewolf, would you be willing to cease "yelling" in this thread? i'd like to have a civil discussion.
peace,
david
@besonen:
the burden of identifying the ddos'd IP(s) could be placed on Linode. then all that Hurricane Electric (HE) would have to do would be to set the null route.
This still requires HE NOC staff to reconfigure their routers, which costs Linode money. The process as you describe it can't be completely automated because Linode's decision to null route needs to be implemented in HE's routing tables.
All you have done is replace a Linode staffer getting a pager notification, investigating and then emailing HE, with a bunch of software that emails HE instead. Given the infrequency of this kind of problem and the time needed to develop the software, this is probably pretty near the bottom of Linode's to do list.
@pclissold:
The process as you describe it can't be completely automated because Linode's decision to null route needs to be implemented in HE's routing tables.
It could be automated. I could think of one obvious way; a web site that linode staff have access to to enter/remove null-routes and an automated process on the HE backend to do that.
It's really not a complicated issue in theory. The practice is harder and would require a LOT of UAT testing on non-live networks before rolling out to production. This is probably more work than HE want to do, even though it's potentially useful for a lot of their customers.
No, the implementation of the null route can be automated on the HE end.
The part I wouldn't want to automate is the decision to null-route a device in the first place! I'd prefer a human to make that decision.
@besonen:
@warewolf:Simple fact of the matter: if you are repeatedly getting DDoS'd, then you are, or have done something wrong. Period.
Simple solution: stop doing what is causing you to get DDoS'd. Period.
you can say that it's a black and white issue, but that doesn't make it so. and it does little to contribute to a thoughtful discussion, imho.
The internet is a hostile environment, rich with people who have the resources to DDoS lowly little Amazon.com, eBay, and Yahoo off the face of the internet. And they've done it. All they need is a reason to do so. Unfortunately the reality of the situation is just that black and white. The gray area is what the definition of "wrong" is. It could be a multitude of things, from being proactive and running an anti-spam service that actually works, running a website that has compromising photograps of someone's significant other, taking over a channel on IRC, taking someone's IRC nickname on a network that doesn't have a "nickserv", proactivly taking down massive botnets that provide DDoS capability, taking down websites that are compromised and serving malicious code that turns Joe Consumer's Unpatched Windows Box into a DDOS zombie, etc etc.
I'm not frustrated at someone inquiring about the way Linode is run, or the policy. I'm explaining why what a lot of people are asking for in this thread is impossible and impractical, and providing a viable and simple solution to prevent DDoS coming your way. Linode is cheap. If you are running a service, or your actions online repeatedly cause DDoS to come your way, then Linode isn't the place for you. That's the point of the three-strike policy. It's an incentive for you to relocate to a service provider (or two, or three) that can provide you with the level of service you require. You need to start shelling out big bucks to buy the WAN pipes that can serve your traffic, and not get saturated by the DDoS traffic you receive.
In the eyes of a Transit ISP (HE.net, l3.net, etc) one man's DDoS is another man's good day of traffic. They simply can't tell the difference. Setting up some kind of automatic system to baseline an IP or netblock's average network utilization will be a maintenance nightmare, and require a lot more interaction with the Service Providers and their Customers. You and I are Customers. Linode is a Service Provider. Hurricane Electric, ThePlanet is a Transit Provider. I'm not blowing smoke, I speak from experience. I am a member of the CSIRC (computer security incident response center) for a US Federal agency that has nearly six hundred thousand public internet IP addresses multi-homed in three separate physical locations through two different transit providers.
It's simple. Don't do things that piss off the people who have the resources to DDoS you off the internet. Your life, and your service provider's lives will be better for it.
@sweh:
@pclissold:The process as you describe it can't be completely automated because Linode's decision to null route needs to be implemented in HE's routing tables.
It could be automated. I could think of one obvious way; a web site that linode staff have access to to enter/remove null-routes and an automated process on the HE backend to do that.
It's really not a complicated issue in theory. The practice is harder and would require a LOT of UAT testing on non-live networks before rolling out to production. This is probably more work than HE want to do, even though it's potentially useful for a lot of their customers.
No, the implementation of the null route can be automated on the HE end.
The part I wouldn't want to automate is the decision to null-route a device in the first place! I'd prefer a human to make that decision.
I would /not/ want my server at a datacenter that had static routing tables editable by anyone but their NOC technicians.
@warewolf:
It's simple. Don't do things that piss off the people who have the resources to DDoS you off the internet.
Well, that is simple. Is there a list somewhere that I can use?
James
@Jay:
I would /not/ want my server at a datacenter that had static routing tables editable by anyone but their NOC technicians.
HE would never allow outsiders to edit their routing tables. The null routing process can't be automated because the people/systems that have to make the decision are not the people who can implement the fix.
JUNOS SDK announcement:
The netname for my node's IP is NETBLK-THEPLANET-BLK-6. ThePlanet will have allocation records of their IPs, but I'm guessing HE would not have a database of that. I'm not even sure how I would tell…traceroute to a router they know is owned by Linode? How would they even decide who is important enough to have in that database? And, would that work if their router is completely overloaded with the DDoS traffic? (I'm honestly curious; is there a 'right' way for HE to identify that an IP owned by ThePlanet is leased to and actively in use by one of their customers?)
Without that, there would be nothing stopping a shady individual working for some company collocated in ThePlanet's facility from plopping someone else's IPs into the null-route table, right?
All of that aside…at least from my experience, getting Level(3) to make a software change is like moving a mountain stone by stone. I would presume HE would be the same way. And my experience has been with very, very minor things. I cannot imagine what it would take to get them to provide an API for null-routing things…
My two cents. shrug