Linode went down 2 days in a row - how can I improve uptime?

Automatic failover is hard. There are lots of ways to do it and they are all complicated and liable to cause downtime themselves if not implemented perfectly. They also vary from service to service. What works for incoming mail won't work for outgoing mail or web. Having two VPS's instead of one also doubles your costs.

It should be far easier to make your Linode more stable. I've seen uptimes of years on linodes as have many other people on this board. The only time I reboot is to resize file systems, upgrade my OS, or take advantage of upgrades.

What's causing the downtime? That should be fixed.

What services do you want to protect anyway?

I was not told what was the exact problem, it was nothing running on my linode and it was out of my hands (the most I was told was there were hardware problems). Each time they shut off my node they make an "emergency ticket" and apologized, it's happened frequently since I purchased my linode this past August/July…

If I cannot determine a good failover method at linode for apache and mysql then I'll likely cancel my account here and go with another company. Mostly the fact that two days in a row there were two issues on my linode really worries me. The Linode support is great, but they can't do anything to guarantee better uptime.

Here's a record of all the downtime:

http://davzy.com/screenshots/Screen_Sho … 122720.png">http://davzy.com/screenshots/ScreenShot2014-01-02at12.26.06_PM-20140102-122720.png

Each of those downtimes were out of my hands (so a problem with Linode itself), with the exception of the one I've put a star by. Whenever it goes down they sometimes offer me $5 credits. $5 is nice but doesn't do anything when the service is offline.

This poor uptime isn't worth the money I pay per month. I've looked at DigitalOcean and a few friends who buy VPSes there have shown that they have better uptime than I've had with linode, and they pay 1/4th of what I pay at linode. I've asked linode if they'll give me a monthly discount and they've refused (which is ironic because if they gave me even $5 a month, I would be willing to throw them an extra $40 for a balancer and second node). I might as well buy several VPSes from DigitalOcean in this case, or go with a company like https://lumous.io/ who offers 100% uptime and then compensation, and when their services go down they (apparently) automatically move nodes to a different server that's online, so that downtime is very small, whereas you can see from my image, on day 2 of downtime with linode I was down for 1 and 1/2 hours.

FYI - Lumous.io (2048M, 2Core, 95G = $108.04/month) is about 2.5x the price of Linode, so 100% uptime guarantee is obviously an expense most people don't need or are unwilling to pay for in a VPS service.

Linode in general has a very good reliability record; it seems that you just got unlucky and were put on an unreliable host machine. I've had between one and three linodes (currently two) for nearly five years now, and I haven't had a single emergency maintenance ticket. That's not to say I haven't had rare unexpected downtime, such as the time the primary and backup power failed at a datacenter, or that time Linode's infrastructure went nuts, but I think there have only been two or three such incidents that have affected me in five years.

@Guspaz:

Linode in general has a very good reliability record; it seems that you just got unlucky and were put on an unreliable host machine. I've had between one and three linodes (currently two) for nearly five years now, and I haven't had a single emergency maintenance ticket. That's not to say I haven't had rare unexpected downtime, such as the time the primary and backup power failed at a datacenter, or that time Linode's infrastructure went nuts, but I think there have only been two or three such incidents that have affected me in five years.

Do you mind telling me where your linodes that have never been down are located? The last Linode could do (I should think) is put me on a service with better uptime, after the uptime disaster I've had to endure.

@vonskippy:

FYI - Lumous.io (2048M, 2Core, 95G = $108.04/month) is about 2.5x the price of Linode, so 100% uptime guarantee is obviously an expense most people don't need or are unwilling to pay for in a VPS service.

If I want constant uptime I would need from linode: 2 linodes and 1 balancer. That's $60 a month and I'm cool paying it. However, I could get a basic managed VPS from Lumous for $40 for the better uptime then that makes more sense, especially considering that I wouldn't need to modify my back end to work clustered, and that I'd get more than $5 back when I experience multiple long downtimes (which would also happen fewer times). That's why I asked for a discount at Linode. After the bad uptime with them, and the cheaper better service elsewhere I would of thought they'd of given me atleast some sort of discount and then I wouldn't of minded spending even more money at linode as opposed to switching, but no dice. If I only ever experienced one downtime with linode I wouldn't be making a fuss, but I've experienced many outages now, 2 outages less than 24 hours apart where one was over an hour is hugely upsetting.

@reditr:

The last Linode could do (I should think) is put me on a service with better uptime, after the uptime disaster I've had to endure.

If this were possible, we'd already be doing it for everybody. If we identify an issue that could cause future problems, we schedule maintenance in the future and alert affected users. If an issue occurs that we could not have seen coming, then we need to do emergency maintenance, as you've experienced. As part of the maintenance, our administrators assess whether we've corrected the issue, and if we believe there will be further instability we migrate Linodes off that host onto other hardware.

Unfortunately, statistically speaking some people will have periods of really bad luck. We do everything we can to detect issues proactively and fix them without affecting Linodes, but that isn't possible for all cases.

• Les

My uptime has been exceptional at Linode just as an FYI.

@akerl:

@reditr:

The last Linode could do (I should think) is put me on a service with better uptime, after the uptime disaster I've had to endure.

If this were possible, we'd already be doing it for everybody. If we identify an issue that could cause future problems, we schedule maintenance in the future and alert affected users. If an issue occurs that we could not have seen coming, then we need to do emergency maintenance, as you've experienced. As part of the maintenance, our administrators assess whether we've corrected the issue, and if we believe there will be further instability we migrate Linodes off that host onto other hardware.

Unfortunately, statistically speaking some people will have periods of really bad luck. We do everything we can to detect issues proactively and fix them without affecting Linodes, but that isn't possible for all cases.

• Les

Even your reply here comes off to me as somewhat of a dodge. You are acknowledging that your service has been poor yet you are implicitly telling me to further endure this poor service. If my "bad luck" is an exception as you say, then you should have no problems offering to move me to a more historically stable server and offering me a slight monthly discount. For one instance of "bad luck" such actions should be extremely minor for you to do. The fact that neither of these things have been done really make me wonder how much linode cares about its customers, or what people think about linode's services in general.

If I pay linode an excess of money in contrast to what I'd be spending at other companies then I expect a better service. Your testimonials are mostly all excellent, and I have no problem paying the linode prices for this excellent service. Statistically since the conception of my account at linode however, I have yet to experience these levels of standards that are worthy of what I'm paying, and so for a subpar service I'd be better suited to switch companies and obtain that same subpar service for a fraction of the cost.

Once again, your technical support are friendly, and I greatly applaud your efforts towards making your service well received by your customers as proven by the excellent testimonials by many of your users. What I have a problem with is that I am an unfortunate example of where your service has been unsatisfactory, and I feel that I have been swept under a rug. The one day one occurrence of 35 minute downtime I could have forgiven, however the 90 minutes of downtime I fell victim to not 24 hours after is both unacceptable due to the repetition of downtime as well as the duration. Especially considering that this second downtime directly followed a previous downtime, and that the time to fix the second downtime was lengthy, I'm not sure why I wasn't immediately contacted and offered to be moved to another node. This would certainly have been a viable option for me especially considering that the size of my node is rather miniscule. But instead, the only solution I was given is that I should be appeased with a $5 coupon and that I throw you guys almost $500 more a year for load balancing to mitigate all this downtime.

Please consider that I have only been your customer for a small number of months, and I have had to endure 4-5 long periods of downtime, when I am paying you 400% more than your competitors, yet the only advice I am given is to spend $500 to implement a failover system myself. I'm not even asking you to price-match your competitors (who also comparatively to my linode have better uptime), but I am asking you to provide me with some solution or even some reason to stay with linode when I'm faced with such indifference to my unreliable experiences here.

@reditr:

Do you mind telling me where your linodes that have never been down are located? The last Linode could do (I should think) is put me on a service with better uptime, after the uptime disaster I've had to endure.

All of these linodes are in Newark. I would also reinforce that it's not the case that I've never had downtime, just that I've never suffered a host-specific failure. I was using the fact that I've had on average two linodes for five years and have never received an emergency maintenance mail to illustrate the point that your downtime has been due to bad luck. That said, Newark has traditionally been a pretty reliable datacentre. Fremont, for its part, had a long period of poor reliability, although I believe Linode switched to a different datacentre in Fremont since those issues.

Your experience is exceptional. I have had a Linode in Fremont for over 10 years. In that time, I have suffered two outages due to power failure and two migrations when host servers began to fail. For you to have had as many troubles as you have in such a short period really is out of the ordinary.

@pclissold:

Your experience is exceptional. I have had a Linode in Fremont for over 10 years. In that time, I have suffered two outages due to power failure and two migrations when host servers began to fail. For you to have had as many troubles as you have in such a short period really is out of the ordinary.

Seconded.

You're bummed at your Linode experience, so move. Random failure is not unique to Linode, so hopefully you'll land on the right side of the bell curve at your next provider - if not, give Linode another shot, just use a different location.

@reditr:

@akerl:

@reditr:

The last Linode could do (I should think) is put me on a service with better uptime, after the uptime disaster I've had to endure.

If this were possible, we'd already be doing it for everybody. If we identify an issue that could cause future problems, we schedule maintenance in the future and alert affected users. If an issue occurs that we could not have seen coming, then we need to do emergency maintenance, as you've experienced. As part of the maintenance, our administrators assess whether we've corrected the issue, and if we believe there will be further instability we migrate Linodes off that host onto other hardware.

Unfortunately, statistically speaking some people will have periods of really bad luck. We do everything we can to detect issues proactively and fix them without affecting Linodes, but that isn't possible for all cases.

• Les

Even your reply here comes off to me as somewhat of a dodge. You are acknowledging that your service has been poor yet you are implicitly telling me to further endure this poor service. If my "bad luck" is an exception as you say, then you should have no problems offering to move me to a more historically stable server and offering me a slight monthly discount. For one instance of "bad luck" such actions should be extremely minor for you to do. The fact that neither of these things have been done really make me wonder how much linode cares about its customers, or what people think about linode's services in general.

If I pay linode an excess of money in contrast to what I'd be spending at other companies then I expect a better service. Your testimonials are mostly all excellent, and I have no problem paying the linode prices for this excellent service. Statistically since the conception of my account at linode however, I have yet to experience these levels of standards that are worthy of what I'm paying, and so for a subpar service I'd be better suited to switch companies and obtain that same subpar service for a fraction of the cost.

Once again, your technical support are friendly, and I greatly applaud your efforts towards making your service well received by your customers as proven by the excellent testimonials by many of your users. What I have a problem with is that I am an unfortunate example of where your service has been unsatisfactory, and I feel that I have been swept under a rug. The one day one occurrence of 35 minute downtime I could have forgiven, however the 90 minutes of downtime I fell victim to not 24 hours after is both unacceptable due to the repetition of downtime as well as the duration. Especially considering that this second downtime directly followed a previous downtime, and that the time to fix the second downtime was lengthy, I'm not sure why I wasn't immediately contacted and offered to be moved to another node. This would certainly have been a viable option for me especially considering that the size of my node is rather miniscule. But instead, the only solution I was given is that I should be appeased with a $5 coupon and that I throw you guys almost $500 more a year for load balancing to mitigate all this downtime.

Please consider that I have only been your customer for a small number of months, and I have had to endure 4-5 long periods of downtime, when I am paying you 400% more than your competitors, yet the only advice I am given is to spend $500 to implement a failover system myself. I'm not even asking you to price-match your competitors (who also comparatively to my linode have better uptime), but I am asking you to provide me with some solution or even some reason to stay with linode when I'm faced with such indifference to my unreliable experiences here.

+1

I think this is a very reasonable perspective that you have here. I'm very interested to see Linode's response to this.

Just an FYI: my experience with Linode has been exemplary; zero downtime.

@reditr:

If my "bad luck" is an exception as you say, then you should have no problems offering to move me to a more historically stable server and offering me a slight monthly discount. For one instance of "bad luck" such actions should be extremely minor for you to do. The fact that neither of these things have been done really make me wonder how much linode cares about its customers, or what people think about linode's services in general.

The point I was attempting to make is that this isn't an effective solution to the problem of unexpected downtime. Because we monitor proactively for potential issues, and correct issues whenever possible before they cause downtime for Linodes, and schedule maintenance where that cannot be done without some downtime, the remaining issues are those which could not be predicted. They are just as likely to affect a "historically stable" host as one that had another issue a few days prior. As such, migrating you to such a host doesn't affect future emergency maintenance.

With regard to reimbursement, we do have an SLA, and we give credit when that SLA is not met. The credit is based on the amount of downtime experienced; we don't provide persistent discounts on the basis of a temporary issue.

• Les

@nickdan:

I think this is a very reasonable perspective that you have here. I'm very interested to see Linode's response to this.
I think it's a stupid perspective.

Run that idea past Target (hey Target, twice now I purchased defective products from your store, please issue me a 5% off card for ALL future purchases or I'll start shopping at Shopko) and see what happens.

If downtime has a negative impact on your business model, then plan (and budget) for it. If your plan is a single host in a single datacenter, then you get what ya get.

@vonskippy:

@nickdan:

I think this is a very reasonable perspective that you have here. I'm very interested to see Linode's response to this.
I think it's a stupid perspective.

Run that idea past Target (hey Target, twice now I purchased defective products from your store, please issue me a 5% off card for ALL future purchases or I'll start shopping at Shopko) and see what happens.

If downtime has a negative impact on your business model, then plan (and budget) for it. If your plan is a single host in a single datacenter, then you get what ya get.

First off your analogy is inherently flawed to begin with because "one bad product = I DEMAND ALL FUTURE THINGS I BUY FROM YOU TO BE CHEAPER" is not the same thing as "this poor service is poor = I shouldn't be paying so much for this poor service".

Second off you completely missed the point. As I previously mentioned I've found several other hosts with statistically better uptime than I've had at linode for 1/4th of the price. I've now already purchased a VPS with Digital Ocean to investigate how reliable they are compared to linode. The thing is, I can buy 2 VPSes from digital ocean and set up a load balancer for half the price of my 1 vps at linode. The same setup at linode would be 3x the price of what I'm already paying linode. I'm doing linode a kind gesture by saying, "statistically your service has been unreliable. I will purchase an additional vps from you and load balancing to correct this issue. You will earn an extra $500 a year from me, so I would like $5 off a month of the service I already have with you that costs me $240 a year". This is certainly me being generous and this is an excellent offer for linode. Even though linode has demonstrated to me that they have an unreliable service, I am willing to spend more money to purchase more services from them, and even with $5 off a month linode is still walking away with one heck of a lot more money than the $10 a month this would cost at another host. Even if I didn't experience all the downtime this is an appealing exchange, so withstanding downtime, the sheer indifference I've encountered from linode is quite surprising.

If linode isn't interested in selling their services, or interested in fixing a customer's service when that service has been irrefutably unreliable, then I'll take my business and my money elsewhere.

@reditr:

@vonskippy:

@nickdan:

I think this is a very reasonable perspective that you have here. I'm very interested to see Linode's response to this.
I think it's a stupid perspective.

Run that idea past Target (hey Target, twice now I purchased defective products from your store, please issue me a 5% off card for ALL future purchases or I'll start shopping at Shopko) and see what happens.

If downtime has a negative impact on your business model, then plan (and budget) for it. If your plan is a single host in a single datacenter, then you get what ya get.

First off your analogy is inherently flawed to begin with because "one bad product = I DEMAND ALL FUTURE THINGS I BUY FROM YOU TO BE CHEAPER" is not the same thing as "this poor service is poor = I shouldn't be paying so much for this poor service".

Second off you completely missed the point. As I previously mentioned I've found several other hosts with statistically better uptime than I've had at linode for 1/4th of the price. I've now already purchased a VPS with Digital Ocean to investigate how reliable they are compared to linode. The thing is, I can buy 2 VPSes from digital ocean and set up a load balancer for half the price of my 1 vps at linode. The same setup at linode would be 3x the price of what I'm already paying linode. I'm doing linode a kind gesture by saying, "statistically your service has been unreliable. I will purchase an additional vps from you and load balancing to correct this issue. You will earn an extra $500 a year from me, so I would like $5 off a month of the service I already have with you that costs me $240 a year". This is certainly me being generous and this is an excellent offer for linode. Even though linode has demonstrated to me that they have an unreliable service, I am willing to spend more money to purchase more services from them, and even with $5 off a month linode is still walking away with one heck of a lot more money than the $10 a month this would cost at another host. Even if I didn't experience all the downtime this is an appealing exchange, so withstanding downtime, the sheer indifference I've encountered from linode is quite surprising.

If linode isn't interested in selling their services, or interested in fixing a customer's service when that service has been irrefutably unreliable, then I'll take my business and my money elsewhere.

Linode works great for me.

@reditr:

Even your reply here comes off to me as somewhat of a dodge. You are acknowledging that your service has been poor yet you are implicitly telling me to further endure this poor service. If my "bad luck" is an exception as you say, then you should have no problems offering to move me to a more historically stable server and offering me a slight monthly discount. For one instance of "bad luck" such actions should be extremely minor for you to do. The fact that neither of these things have been done really make me wonder how much linode cares about its customers, or what people think about linode's services in general.

Few things:
* Les did not dodge anything in his reply to you. He was actually more direct to you than you'll find at many other hosting companies.

• He also made no acknowledgement that Linode's service has been poor and did not imply telling you to do anything, nor did he in fact give you any advice whatsoever.

• You are fighting with Linode because computer hardware fails and are expecting Linode to offer you a discount because computer hardware fails. You might as well ask for a discount for the weather, too.

I have no information on the specific nature of your case but, often, these emergency maintenance procedures rectify a situation on the system that could lead to even worse problems than having your Linode rebooted.

@reditr:

If I pay linode an excess of money in contrast to what I'd be spending at other companies then I expect a better service.
Take four six-figure salaries for a developer in the Bay Area. Add them together.

I've been an on-call administrator on an Amazon account that costs that much every month, and a significant portion thereof was dedicated to premium support. We had several hundred instances unreachable for over eight hours and our tickets sat unanswered for all of that time, and nobody answered the only phone number we had. You had one instance's host machine fail twice, Linode was communicative with you, and you expect Linode to bend over backwards for you. Amazon did not even credit us.

Your assertion that support quality has any relationship with account spend is, as demonstrated, flawed. Linode is known among operations people as having among the best support in the industry. DigitalOcean doesn't wipe their drives when you delete an instance and, when this gaping security hole is reported to them, goes after the person reporting the issue as being irresponsible. Good luck with what you pay for at DigitalOcean and consider all sensitive information on your "droplet" as immediately compromised.

@reditr:

Please consider that I have only been your customer for a small number of months, and I have had to endure 4-5 long periods of downtime,
I thought it was two. I ruled out your diagram as useless when it had a four month outage in it.

@reditr:

when I am paying you 400% more than your competitors
You are not. DigitalOcean is the gas station burrito shelf, Linode is Chipotle. They are not in competition with each other, though one likes to think they are.

/me grins at Jed being able to say it like it is now that he no longer works for Linode