Linode Backup Service (beta)
caker
Linode Staff
If you're participating in the beta and discover an issue, please open a new thread.
We want you to have backups. Hell, I want backups. But they're tedious to configure and monitor. We've all heard the stories (and possibly experienced them) when you need a backup long after configuring them you realize they haven't been running successfully for months. So, our goal for the Linode Backup Service was ambitious: create a reliable, redundant and highly available system, that's easy to use (set-it-and-forget-it), affordable, and Just Works.
Description
The Linode Backup System is designed to be an easy to use, reliable and redundant on-site backup solution for your Linode. It performs backups without causing any interruption of your running system, and is seamlessly integrated into the Linode Manager.
Backing Up
There are four backup slots: Three of the slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. The fourth backup slot is a user-initiated snapshot and remains in place until another user-initiated snapshot is taken.
You can configure the time upon which the automatic backups are initiated from a list of 2 hour windows – you'll want to perform any database dumps before this window. You can also configure which day of the week to consider for the weeklies.
Restoring
You can restore a backup to any of the Linodes attached to your account, even if it does not have backups enabled. Currently only a full restore is possible.
Redundancy
We built a custom distributed and scalable storage engine which will replicate your files across at least two storage nodes. Even your backups are backed up!
Features and Limitations
The backup system must be able to mount your disk images on the host. If you've used fdisk on your images to create partitions, or created encrypted volumes, or done anything other than use our deployment or disk image creation tools, we won't be able to back up the data. The backup system operates on files, not at the block level.
A failed backup will never rotate out a good one. If a backup fails on the day of a weekly backup, the next oldest backup will be used for that weekly slot.
Files that have been modified, but are the same length and without any metadata changes (like mtime) will not be considered "changed" during a subsequent incremental backup.
Currently, only ext2/3 volumes can be backed up. This limitation will be removed in an upcoming release.
ACLs are correctly backed up and restored.
Pricing Structure (tentative)
Linode 360: $5.00/mo
Linode 540: $7.50/mo
Linode 720: $10.00/mo
Linode 1080: $15.00/mo
Linode 1440: $20.00/mo
Linode 2880: $40.00/mo
How do I participate in the Backup Service Beta?
Participating in the beta is free of charge.
We've only deployed the storage hardware in the Newark, NJ facility. So, for now, you must have a Linode in Newark to participate in the Backup Service Beta. Open a ticket under that Linode and request that we enable backups.
What happens at the end of the beta?
The backup service will be scheduled to be turned off automatically. However, we'll give everyone the chance to retain their backup service and will start charging for it.
What kind of risk is there? Is this thing going to work?
There's little risk to your existing data. However, I wouldn't rely on our backup service as your only form of backup just yet. During the beta period, backups may be inconsistent or incomplete, or we may need to wipe all of the backup data and start fresh. In other words, keep your own backups and don't rely on this thing working. You've been warned!
What about the other facilities?
Atlanta and Dallas sometime later this month, and Fremont early May.
-Chris
44 Replies
Also, will the backups be available outside of the linode network - i.e. can we access them from home?
@caker:
Files that have been modified, but are the same length and without any metedata changes (like mtime) will not be considered "changed" during a subsequent incremental backup.
This may not work very well for things like MySQL InnoDB tables, which allocate chunks of disk. I realize a database is not the best example, but let's say it's a big, mostly read-only database.
@saman007uk:
Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.
It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.
@jcr:
Great idea!
Any idea of how long a backup should take?
I'm backing-up my Linode 360 atm. So far it's taken 1.25 hours.
caker
Linode Staff
@phvt:
@caker:Files that have been modified, but are the same length and without any metedata changes (like mtime) will not be considered "changed" during a subsequent incremental backup.
This may not work very well for things like MySQL InnoDB tables, which allocate chunks of disk. I realize a database is not the best example, but let's say it's a big, mostly read-only database.
This will work fine because the mtime changes every time a change is made.
-Chris
caker
Linode Staff
@phvt:
It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.
I beg to differ:) . This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.
@saman007uk:
Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.
Doing a snapshot does NOT always mean you're doing to get your mysql DB files in a consistent state. It can depend on which type of table engine you're using, for instance…
So dump your DBs to a file before the backup window just like you would with any other backup configuration.
-Chris
@caker:
I beg to differ
:) . This is a very sophisticated system which does involve LVM snapshots on the host.
@caker:
I beg to differ
:) . This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.
My understanding is that LVM snapshots can be made of running InnoDB tables and restored successfully. The InnoDB tables will be in "crashed" state, but the data will recover automatically.
There's also the option to mysqldump –single-transaction with InnoDB and get a consistent backup without blocking applications.
Other MySQL storage formats (like the default MyISAM) don't have this ability.
By "simple" I didn't mean simplistic, I meant like K.I.S.S.!
The first is taken care of by the user-selectable backup, but will the second be available? I.E. Will I ever be able to grab a disk image?
…Chris
caker
Linode Staff
@phvt:
@caker:I beg to differ
:) . This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.
My understanding is that LVM snapshots can be made of running InnoDB tables and restored successfully. The InnoDB tables will be in "crashed" state, but the data will recover automatically.
Yeah, but there's a potential snag when involving a virtualization layer:
We're issuing the LVM snapshot on the host – not from within your Linode's kernel. Performing a snapshot flushes dirty buffers, but since that's being done on the host your Linode kernel is unaware that this is happening, so it still has dirty buffers that were never written to disk.
This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file…
-Chris
caker
Linode Staff
@xoff00:
The first is taken care of by the user-selectable backup, but will the second be available? I.E. Will I ever be able to grab a disk image
Restoring restores to disk images, which you can then mount and pull down on your own with whatever (rsync, scp, tarball, etc).
-Chris
caker
Linode Staff
@zengei:
Just making it clear: ext4 disk images are not currently supported, correct?
They will be very soon – maybe a few days.
-Chris
@caker:
@zengei:Just making it clear: ext4 disk images are not currently supported, correct?
They will be very soon – maybe a few days.-Chris
Great, look forward to it.
@caker:
Yeah, but there's a potential snag when involving a virtualization layer:
We're issuing the LVM snapshot on the host – not from within your Linode's kernel. Performing a snapshot flushes dirty buffers, but since that's being done on the host your Linode kernel is unaware that this is happening, so it still has dirty buffers that were never written to disk.
This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file…
Thanks–this is the detailed explanation I was looking for--how these snapshots are different from direct LVM snapshots, and why a db dump is very much necessary.
I will definitely purchase this option whenever it becomes available!!!
Jeff
snapshot 2009-04-03 21:07:16 2009-04-03 21:10:42 3 minutes, 26 seconds successful
snapshot 2009-04-03 14:21:03 2009-04-03 19:33:48 5 hours, 12 minutes, 45 seconds failed
@saman007uk:
Just tried to backup my 360. I'm only using 1.9GB, and the backup took 5 hours!
I'm not surprised at the failures – I've been taking things up and down all day, making improvements for speed.
-Chris
@saman007uk:
Just tried to backup my 360. I'm only using 1.9GB, and the backup took 5 hours!
Didn't you see the note that said the backup server is connected via dial-up from caker's basement?;)
I say his because my on-demand backup failed; you were probably tweaking
If that is the case, would you consider pricing by size rather then node type? Because if i could only backup say my root non lvm image, im not really able to back up all that much data..
@Internat:
Just checking, but if i use paritions on a drive.. or more to the point put a "drive" into an LVM.. and then parition on top of that im going to assume im not going to be able to use this backup option correct?
All I could think up for this situation so far is to make a snapshot, mount a "normal" disk image and then rsync the snapshot to that, prior to the backup window. But now you've made your own backup (using more expensive disk space)…
@caker:
This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file… It would be nifty to expose this functionality to the API.
:)
@caker:
@phvt:It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.
I beg to differ:) . This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.
@saman007uk:Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.
Doing a snapshot does NOT always mean you're doing to get your mysql DB files in a consistent state. It can depend on which type of table engine you're using, for instance…So dump your DBs to a file before the backup window just like you would with any other backup configuration.
-Chris File databases like sqlite3 needs special care (dump before backup) like mysql, or they just get backuped without problem?
@Andrea:
File databases like sqlite3 needs special care (dump before backup) like mysql, or they just get backuped without problem?
Just put a short script into your crontab to dump your DB every day into an sql file or something just before the backup. This is all special care you need!
@RossH:
Well, I have a suggestions if it isn't already implemented, the ability to mount backup images so you can pull single file backups off of them.
I second the above idea. This would introduce amazing flexibility into the system and possibly prevent unnecessary downtime. Is there anything like this yet?
and "duplicate image" in my dashboard?
With duplicate image I can create a really perfect backup,
so how this service can improve the old one?
> Especially after the Dallas incident (which resulted in total data loss) backups became more important.
What do you mean "more important"? I find backups have always been important. You should have them, make them, test them.
Did you not have backups of your stuff?
I don't know if you were backing your data up or not, but here is my opinion on the subject:
To quote myself (
> This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.
Even if/when Linode gets backups working, you are still responsible for your data. I'll subscribe to their backup plan (depending on price point) for convenience, but I'll also be doing my own backups independently or downloading my backups from Linode (if that's possible) and testing them to make sure the data is there and uncorrupted. Which is why I want to get on the beta when it becomes available again so I can see what it's all about and how it works.
When I started with linode, a linode64 came with 4.5Gb of disk (if you paid yearly and so got 1.5Gb extra free). Now the minimum comes with 16Gb disk (or 24Gb if grandfathered in with old bonus plan). I've only allocated 60% of the disk space I have, so far!
This means I have plenty of backup space. In this I do a weekly level 0, daily incrementals. And these get rsync'd daily to my home machine.
So if things totally utterly exploded I'd need to recreate disk profiles, boot into finix, scp up the backup files, restore data. Of that the scp would probably be slowest since my upload speed from home is only 5Mbit/s and my level 0 gzip'd files come in around 2Gbytes in total.
Of course, I also run an essentially mirrored configuration at Panix, so I'd just have to modify DNS to point there while I was rebuilding.
@caker:
What about the other facilities?
Atlanta and Dallas sometime later this month, and Fremont early May.
-Chris
Hello,
Are these available in Fremont now, I'll open a ticket if that's the case.
@kornrunner:
Is (beta) backup feature in plan for London datacenter?
From what I've heard: Yes,, the next version of the backup system will be available in London as well. There is no ETA as to when that will happen.