Linode Critical Xen Maintenance

There is a 2 hours window allocated.

For such a global upgrade, can we expect the reboot to happen at the very beginning of that 2 hours window, based on your experience?

What Linode normally do is sometime at the start of the window all nodes on the hosts will be shutdown. Then they'll patch the host, reboot it and start bringing up the nodes. Nodes are staggered when booting up so how long your downtime will be will depend on where you land in the queue.

I don't work for Linode so can't say that's what will happen but it's an educated guess ;) Linode are very good at handling these issues so I wouldn't worry just be patient and have a read of this https://www.linode.com/docs/uptime/rebo … ival-guide">https://www.linode.com/docs/uptime/reboot-survival-guide

Related question: How do we ensure that our machines will be "cleanly shut down" from the outside?

When using Debian under KVM and virt-manager at home, I need the acpi-support-base and acpid packages to be installed so that the "shutdown" menu works. Surprisingly, this package was not installed by default on my first linode Debian 7 deployment. Is Xen different about this?

@sanvila:

Related question: How do we ensure that our machines will be "cleanly shut down" from the outside?
The mechanism should be identical to when you use the Shut down or Reboot buttons in the Linode Manager (or the automated shutdowns during a migration or upgrade, for that matter). So if you wanted, you could run a test in advance when convenient on your own schedule to ensure your Linode will do the right thing during such a shutdown/reboot cycle. But yes, for me such shutdowns (whether manually by me or automatically by Linode) has always been a graceful shutdown assuming the guest was operating properly in the first place.

> When using Debian under KVM and virt-manager at home, I need the acpi-support-base and acpid packages to be installed so that the "shutdown" menu works. Surprisingly, this package was not installed by default on my first linode Debian 7 deployment. Is Xen different about this?
I'm guessing here, as I don't know for sure, but Linode uses Xen PV for its guests, so I expect the PV drivers are shutting down the guest from within the guest itself. It's not simulating an external hardware signal to generic drivers; rather the Xen aware drivers execute a shutdown on your behalf.

– David

We send a control-alt-delete (CAD) signal to your init process - which, in all distros I've ever seen, is configured to begin a shut down. In the case a Linode is misbehaving or doesn't exit after 2.5 minutes, we then issue a flush/sync and pull the virtual plug.

-Chris

Really just a comment….

It is interesting that the notice in the Linode Manager is less than useful….

Nice to know what my default timezone is set to, but what is the actual time of the outage?

Just received notice via email, which lists the times in UTC. So what is the point of telling me my default timezone if the notification is using UTC anyway?

One of my Linodes is getting less than 24 hour notice. That is a bit unusual. The other is 5 days out, which makes it easier to notify clients involved.

The time is different for each Linode, and as per the big yellow banner, each will be worked on "at the time listed below".

Look in the table of Linodes?

• Les

@akerl:

The time is different for each Linode, and as per the big yellow banner, each will be worked on "at the time listed below".

Look in the table of Linodes?

• Les

don't mind me… I am blind. I didn't notice the extra column maint times in the list of linode for some reason…

Can anyone who has had the maintenance done to their Linode comment on how long the downtime lasted? I understand it's scheduled for 2 hours but the actual downtime can be much less. Thanks.

@kcristella:

Can anyone who has had the maintenance done to their Linode comment on how long the downtime lasted? I understand it's scheduled for 2 hours but the actual downtime can be much less. Thanks.
For me this morning 9am MST, it was about 30-35 mins. I brought down my Linode myself and powered it up when the dashboard said complete rather than letting them shut it down and bring it back up. Might have been a bit faster as I wasn't watching the dashboard every minute.

I've had one so far, it was down 15 minutes. Fully automatic and no problems.

@obs:

I've had one so far, it was down 15 minutes. Fully automatic and no problems.

Good to hear, thanks for the info!

Here goes!!

All appears okay and back up 45 minutes after a brief message about a hardware failure during maintenance.

I'm pretty sure the physical hardware issue alert is normal. I guess a monitoring system sees that the host is down, doesn't know that it's because of expected scheduled maintenance, and throws up a generic message.

Well. The email I got says my linode will be rebooted on 2015-03-07 12:00:00 AM UTC

So what hour is that? I generally manage to figure out the AM/PM nonsense by thinking 'AM=after midnight' (yes I know it's meant to be some Latin). But 12:00:00 AM? Is it midnight or is it noon? Even a web search indicates it's ambiguous.

I'm sorted now because I just logged in to the linode manager and it says it's 11 hours left.

But come on, why use such an unscientific, stone-age time system? It starts out fine with '2015-03-07', standard date format and all, and then runs off into something that could as well be furlongs. Please, this is the 21th century, use 24 hour time formats!

12:00:00 AM is 00:00:00

As I said, I figured that out already due to the '11 hours left' message. The point is, why send out emails with a time specified in that horrid am/pm format? Most countries (nearly everyone) in the world are not in am/pm land, and use a sane 24 hour clock. Likewise, Linode's customers are all over the world. Linode should be precise with such important information, and that means using a sane time format in their email communication. And no, that has nothing to do with language. My work is 100% international, nearly everybody communicates in English, and of course everybody uses a 24-hour time format independent on which language is in use.

@T-A:

of course everybody uses a 24-hour time format independent on which language is in use.
s/everybody/almost nobody/

There, fixed that for you.

If Linode sent out an email saying the maintenance started at 03:00:00 then people would ask "is that am or pm?" Even though I speak 24hr, doesn't mean the person sending to me does!

The answer, of course, is "00:00:01 am" and "12:00:01 pm"; always add that extra second :-)

I don't think so. If they sent a message saying the maintenance starts at "2015-03-06 03:00:00", people could easily infer from the year-month-day order that they are using the ISO format for the entire timestamp, not just for the day.

The ISO format does not have AM/PM. See http://en.wikipedia.org/wiki/ISO_8601 for details.

I also had to think twice about the hour when I saw "12:00 AM" so the argument that they somehow "have" to use AM/PM to reduce confusion does not hold, IMHO.

@T-A:

As I said, I figured that out already due to the '11 hours left' message. The point is, why send out emails with a time specified in that horrid am/pm format? Most countries (nearly everyone) in the world are not in am/pm land, and use a sane 24 hour clock. Likewise, Linode's customers are all over the world. Linode should be precise with such important information, and that means using a sane time format in their email communication. And no, that has nothing to do with language. My work is 100% international, nearly everybody communicates in English, and of course everybody uses a 24-hour time format independent on which language is in use.

Linode is an American company, Americans use AM/PM. I work for an enormous international company based in NY. Almost everything that comes out is stated in AM/PM and in ET (US/Eastern time). Everyone else knows how to convert it.

A google is only ambiguous because some pedantic moron has butchered wikipedia to make it so. 12:00 AM is midnight, period.

As a random aside: a.m. is "ante meridiem", or before midday/noon, and p.m. is "post meridiem", or after midday/noon. So, conventions aside, "12:00 am" would logically be when the hour hand points at the '12' prior to when the sun passes closest to your local zenith on a given day. "12:00 pm" is still a crapshoot, of course.

Good news is that there's a nice countdown timer when you look at the overview page for your Linodes, so you can calibrate accordingly.

Easy way to keep it straight:

1am is in the middle of the night, therefore, 12am (1 hour before) is midnight.

1pm is the middle of the day, therefore, 12pm (1 hour before) is noon.

It's not all that hard, but I agree, 24hr time is much clearer.

By that logic, "11:59:59am is the middle of the day, so 12:00:00am should be the middle of day".

all done in ~30 mins for me

Almost an hour for me for a Linode 2 GB (the $20/mo plan). The reboot seems to be hung for the past 10 minutes though. I'll wait a bit longer, the host could just be busy.

Finished in less than 20 minutes for a Linode 1GB, nice work!

@sanvila:

I don't think so. If they sent a message saying the maintenance starts at "2015-03-06 03:00:00", people could easily infer from the year-month-day order that they are using the ISO format for the entire timestamp, not just for the day.

Except for those who don't know about such things.

Using AM/PM removes any ambiguity as to what's what.