Linode Backup Service (beta 2.0)

Awesome! If we were in the original backup beta program, will be automatically be enrolled in this one? Checking my dashboard… it would appear to be the case, but I wanted to verify.

Thanks!!

@jcw5002:

Awesome! If we were in the original backup beta program, will be automatically be enrolled in this one? Checking my dashboard… it would appear to be the case, but I wanted to verify. Yes…

are there any known issues that we should be aware of?

@wjstone:

are there any known issues that we should be aware of?
None that I didn't outline above.

-Chris

If there's room in the cool club, can I get in? :roll:

@jstn:

If there's room in the cool club, can I get in? :roll:
There's a LOT of capacity with our current deployments. Just submit the ticket above and you'll be set!

-Chris

I set it up on two of my linodes (both 540s). Here are the times:

Disk utilization:

Linode 1: 1.7GB

Linode 2: 13GB

Initial snapshot

Linode 1: 4m18s

Linode 2: 1h49m36s

First automatic snapshot (6h-8h)

Linode 1: 1m11s

Linode 2: 10m54s

I'm not sure why it's so slow on linode 2; sure, it has a lot more data than linode 1, but 11 minutes is slower than my own rsync-based backups over the internet (which cover the vast majority of the filesystem), and the initial snapshot is 20x slower than Linode 1 for only 8x the data.

Is this normal? Higher load on linode 2's host? I don't think it's necessarily a problem, but that initial snapshot very nearly went over the 2 hour window mark had it been an automatic.

Guspaz, it's because it's doing far more awesome than you can possibly realize. Nothing to be concerned about.

-Chris

@caker:

Guspaz, it's because it's doing far more awesome than you can possibly realize. Nothing to be concerned about.

-Chris
What sort of host impact can be expected by ramping up the backup beta again (or later in production)? One of my Linodes gets fairly serious I/O wait times (it itself does very little I/O) generally in the early AM hours ET. I manually execute some regular tasks in that time block each day so experience the differences. Strangely it's been way better recently until the last few nights.

Definitely not scientific but the performance degradation seemed inversely correlated to the interruption in the backup beta, and since I suspect many folks would prefer off hours backups (it's a Newark Linode) it's probably not unreasonable to think that some peer Linodes might be in the beta and getting backed up in that window. Of course, it could just be another Linode who started nightly processing, or something end-of-month related, etc… But it does make me wonder about the additional I/O the backup process must by necessity add to the host system.

Given that disk I/O is a constrained resource, what sort of additional I/O overhead from the backups running are anticipated on a given host? I'm assuming snapshots/backups are spread across time to avoid spikes, but is there a worst case analysis if all Linodes on a host are participating or if a large number choose the same time window?

-- David

Edit: Forgot to mention that my Linode itself is not in the backup program. I realize that running on top of a snapshot during a Linode's own backup might have different costs - I'm more interested in the potential backup impact on other Linodes on the same host.

db3l, that is something that's very difficult to answer - it depends on many factors. But in the vast majority of cases it will have little to no noticeable impact.

We schedule backups on hosts to prefer only one is running at any given time, to avoid backup-storms as best as possible. During a backup an LVM snapshot is created, and during this time writes from within the Linode that's being backed up do get amplified into a few io ops, but whether this has any noticeable effect on the host overall depends on the activity of those nodes at the time, and what type of IO they're doing. Backup IO is no different than Linode IO, and our same fair-queuing and io-priority facilities apply.

Keep in mind that after the initial backup, subsequent backups are light weight - a simple filesystem scan and then reads of any files that have changed, and then a scrubbing of the cow file that the snapshot has generated (which itself is throttled against io load on the host).

If I found the correct account, one of your Linodes is indeed participating in the backup beta, however the window is set to 8:00 AM eastern. No users on that host have a backup window in the early AM.

-Chris

@caker:

We schedule backups on hosts to prefer only one is running at any given time, to avoid backup-storms as best as possible. During a backup an LVM snapshot is created, and during this time writes from within the Linode that's being backed up do get amplified into a few io ops, but whether this has any noticeable effect on the host overall depends on the activity of those nodes at the time, and what type of IO they're doing. Backup IO is no different than Linode IO, and our same fair-queuing and io-priority facilities apply.
Knowing that you try to prefer one at a time is very helpful to know, thanks. So it's additional I/O that's not being generated by peer Linodes themselves, but does sound like it shouldn't cause any real impact if it's playing by the same rules as the Linodes themselves for I/O.

> If I found the correct account, one of your Linodes is indeed participating in the backup beta, however the window is set to 8:00 AM eastern. No users on that host have a backup window in the early AM.
Oops, I should probably have mentioned - the Linode in question is on a business account since it uses separate billing, while I use a personal account (which yes, has a Linode on the beta) for forums.

The questionable Linode is on host newark174 and my morning processing is typically in the midnight-2am ET range. Tuesday morning (~1am) I got average I/O waits in the 30-50% range with peaks at 100% for a bit, so a database transfer that typically takes a few seconds probably took almost a minute (small absolute, huge relative change). I'd normally just put it up to host contention but couldn't help noticing the timing with the backup beta, given how good the I/O performance had been recently. Maybe others on the host are just getting their first full backups or something, but given your description, it sounds more likely that it's just coincidence. If it continues I'll worry about digging further, but that's definitely off topic for the backup beta.

Thanks.

– David

Here's my $0.02.

My 360 node took 6 mins 51 seconds to do the initial one and 1 minute 40 seconds to do a snapshot, so all good :)

One question though, the snapshots are the full backups or incremental? Looking at the time I'd say incremental, which raises the question what happens when the auto ones become newer than the snapshot?

@obs:

One question though, the snapshots are the full backups or incremental? Looking at the time I'd say incremental, which raises the question what happens when the auto ones become newer than the snapshot?
Don't think about it like that. We do all sorts of magic that handles rotating out unreferenced files, etc. A snapshot is a full snapshot. The autos rotate out.

-Chris

Nice :) I like magic.

I'll probably try creating a new node sometime soon and restoring to it to see how it goes.

Out of curiosity how long do you plan on testing for? (Assuming nothing screws up).

We think it's production ready, but wanted to widen the testing to make sure there aren't any issues that would be more easily corrected under a beta flag than a production one. I don't like to give out ETAs, but it's my hope to launch the service within the next few weeks, barring no larger issues arise.

-Chris

Any news about when the backup service will be available for servers in the London datacentre?

@k33l0r:

Any news about when the backup service will be available for servers in the London datacentre?
Did you make it as far as the second sentence of my initial forum post? :)

-Chris

@caker:

@k33l0r:

Any news about when the backup service will be available for servers in the London datacentre?
Did you make it as far as the second sentence of my initial forum post? :)

-Chris

Sorry, I only asked because the last time I brought this up I was told that the backup service wasn't available in London, not even as a beta level service.

@caker:

There is no cost involved (it's free!) and Backup Service Beta is available in all facilities.
Hello - can you clarify whether it is only free while in beta, or will it remain a free service once in production?

@lennox:

Hello - can you clarify whether it is only free while in beta, or will it remain a free service once in production?
Free while in beta. Cost to be determined.

-Chris

I'm assuming it'll be okay since it works at the file and not block level but I just want to clarify: if the partition is resized will backups from before the resize still be usable?

@amarshall:

I'm assuming it'll be okay since it works at the file and not block level but I just want to clarify: if the partition is resized will backups from before the resize still be usable? Yup.

-Chris

I just want to say that my backup has been working flawlessly since it was tweaked last time by the developers. Great job guys! Looking forward to signing up for your backup service. Any idea ref. cost or still trying to determine it?