Load Balancing across datacenters

We are looking to accomplish something similar for two of our clients. I think I'm going to use a small VPS in "location 3" to check the status of the server in "location 1" every minute and use the Linode API to change the DNS of the domains to "location 2" if the location 1 server isn't available.

Just to note, DNS failover isn't that reliable, you have to hope that whatever DNS resolver the client is using respects TTLs (ISPs often ignore them) and some browsers don't always obey TTLS (firefox for example requires a restart).

obs,

Luckily for me both clients are rather small and I can dictate software & configurations. AFA DNS failover, I'm going to have to test what works best. I'm hoping that if I use Linode's DNS servers that I can use the API to change IP's and that I can have very low TTLs.

I did find that Rackspace load balancers support any IP address. Wondering if anyone else has other suggestions.

MSJ if you use Linode's nameservers be aware that they only update every quarter hour so if the api call hits at 00:01 you'll have to wait 14 minutes for it to take effect regardless of the TTL.

ballagas, I maybe wrong but I believe that rackspace's load balancers don't fail across DCs so you'd still have a single point of failure in that the loadbalancer is in one DC.

To achieve a decent cross-dc failover system you need to be able to redirect your IPs to different servers, services like cloudflare and incapsula can do this but I find their stability (or lack of) isn't worth it. It has been several months since I last tested them so YMMV.

obs, thanks for the tip. A 14:59 possibility is not what I'm looking for.

I'll have to look into redirecting manually (perhaps using a subdomain?), or some other method for making this type of magic happen without updating the DNS. I'll have to figure out a way to do it without a big performance hit.

From the DNS front, I've had good luck with DNSMadeEasy in the past, and the price is comparatively trivial. I'd take a look at their stuff for DNS failover before rolling my own. (There are numerous other providers of this service, they're just the ones I've used.)

Does anyone have experience with CDNs for fail-over load balancing, something like Fastly (https://www.fastly.com/) or Cedexis (http://www.cedexis.com/)

It would have to be a DNS solution - one that will allow a healthcheck on a target and the ability to then automatically switch DNS to a healthy target.

Amazon has this in their Route53 DNS.