High availability

@xerbutter:

1) If the linodes are in separate datacenters, is there any single point of failure associated with linode? Is it any more likely that both linodes would go down at the same time than 1 linode and 1 server at another host?

2) Is there any data on uptime in each datacenter? Or any way to get a sense for what the probability would be that both servers would be down at the same time.

I can't meaningfully answer #1 due to lack of detailed information about the entire setup. If nothing else, divergent network circuits leaving the data center often goes through the same physical paths (since it's typically the same ILEC doing the last mile wiring – or different LECs/providers running their cabling through the same physical conduits, or even a LEC/provider contracting with another to carry their last mile traffic). True path diversity often won't be guaranteed by the telcos - and is not typically disclosed to customers. (I say this based on some work experience at a telco.)

So what that means is, even if you had multiple network providers within the same data center, you have no hard guarantees that a backhoe cut won't cause them all (or just yours) to lose service, or cause an overload condition that effectively blocks your service.

But with that said, Linode and the data center generally has redundancy built into just about every level of the operation.

As for #2… I'm not sure if Linode tracks uptime. But IIRC, datacenters typically promise about 5 nines (99.999% uptime). They usually reach it unless there's been a lengthy power outage, which is not common but yet not unheard of, either. I've been here for.. oh, I don't know… nearing 4 years?

In that time, I think I've had two datacenter-wide power outages -- each outage taking between 2-4 to 8? hours to fully resolve. And then I've had a a handful of stupid people DDoS'ing another Linode in the same datacenter, which takes out everybody's network service within the same data center. And on a few occasions, an user was thrashing the host due to inefficient use of memory or disk I/O -- but for each, the staff quickly stopped it and had the issue corrected promptly by the other Linode owner on the same host.

So my answer is: it's generally extremely unlikely for hosts at two separate datacenters (each datacenter having its own power plant, power infrastructure, different ILECs, redundant routers, different network providers, different geographic locations, and so forth) to be knocked offline concurrently. The only exception is if someone with a major beef against Linode DDOSes all three datacenters concurrently -- which is not something I have ever seen during my time here.

Unlike most folks who has servers with the same provider and on the same network, Linode uses multiple networks/providers at each data center, so any network issue affecting one host in a datacenter rarely affects any other hosts in other data centers. That's a very good thing(tm).

@tronic:

As for #2… I'm not sure if Linode tracks uptime. But IIRC, datacenters typically promise about 5 nines (99.999% uptime).

That's actually not the case. If you have five nines, you can only be down for ~30s per year. This is considered "carrier grade" and costs a bunch of money.

Most reasonable datacenters promise something like 99.9% (8.5hrs/yr) or 99.95 (4.5hrs/yr), and sometimes will promise higher uptime for specific things; i.e. power. (99.95% total uptime, 100% guaranteed power uptime is the SLA for the datacenter my company uses).

@Jay:

That's actually not the case. If you have five nines, you can only be down for ~30s per year. This is considered "carrier grade" and costs a bunch of money.

Most reasonable datacenters promise something like 99.9% (8.5hrs/yr) or 99.95 (4.5hrs/yr), and sometimes will promise higher uptime for specific things; i.e. power. (99.95% total uptime, 100% guaranteed power uptime is the SLA for the datacenter my company uses).

Ahh, yes, you're right – thanks. I do recall that at least one of the Linode datacenters promised an availability that took just one disaster to bust. I had a Linode at ThePlanet in Dallas when they had a massive outage a few years ago. While waiting for it to come back up, I had time to find their corporate webpages talking about promised SLAs -- was good for a chuckle. With that said, Linode's datacenter outages are very few and far between.

Thanks for your thoughts. So it sounds like there are 2 ways that having 2 linodes in different data centers could be more vulnerable than 1 linode and 1 other host:

1) Someone targets linode for an attack and DDOS's both linode data centers.

2) Something about linode's architecture has a single point of failure (something about lish or account management…something…) but we're not sure if this vulnerability exists.

On the positive side, managing 2 or even 3 linodes would be a lot easier with the cloning tools, etc. That might help minimize any errors on my part in making sure each server is setup correctly.

Thanks for your uptime report also. I'm not quite sure how to estimate the probability that 2 linodes vs 3 linodes would be down…so if anyone has any experience with that, I'd love to hear about it!

Just to share a bit of experience. I have two linodes, one in Dallas, one in Atlanta with a failover with dnsmadeeasy. Over the last six months, the failover has kicked in once in a while (I don't keep statistics of those events) but I have never experienced Dallas and Atland being down at the same time. Concerning the uptime data, I have always wondered if some packet losses is considered "down time" or "uptime".