Linode Backup Service (beta)

Yay Caker you've made my day - thanks a lot ! :D

Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.

Also, will the backups be available outside of the linode network - i.e. can we access them from home?

Awesome, thanks! Looking forward to checking it out! And I'm glad it's in Newark :)

@caker:

Files that have been modified, but are the same length and without any metedata changes (like mtime) will not be considered "changed" during a subsequent incremental backup.

This may not work very well for things like MySQL InnoDB tables, which allocate chunks of disk. I realize a database is not the best example, but let's say it's a big, mostly read-only database.

@saman007uk:

Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.

It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.

Great idea!

Any idea of how long a backup should take?

@jcr:

Great idea!

Any idea of how long a backup should take?

I'm backing-up my Linode 360 atm. So far it's taken 1.25 hours.

@phvt:

@caker:

Files that have been modified, but are the same length and without any metedata changes (like mtime) will not be considered "changed" during a subsequent incremental backup.

This may not work very well for things like MySQL InnoDB tables, which allocate chunks of disk. I realize a database is not the best example, but let's say it's a big, mostly read-only database.
This will work fine because the mtime changes every time a change is made.

-Chris

@phvt:

It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.
I beg to differ :). This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.

@saman007uk:

Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.
Doing a snapshot does NOT always mean you're doing to get your mysql DB files in a consistent state. It can depend on which type of table engine you're using, for instance…

So dump your DBs to a file before the backup window just like you would with any other backup configuration.

-Chris

@caker:

I beg to differ :). This is a very sophisticated system which does involve LVM snapshots on the host.

:D :D :D I like "very sophisticated" here!

Just making it clear: ext4 disk images are not currently supported, correct?

@caker:

I beg to differ :). This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.

My understanding is that LVM snapshots can be made of running InnoDB tables and restored successfully. The InnoDB tables will be in "crashed" state, but the data will recover automatically.

There's also the option to mysqldump –single-transaction with InnoDB and get a consistent backup without blocking applications.

http://dev.mysql.com/doc/refman/5.1/en/ … ransaction">http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#optionmysqldumpsingle-transaction

Other MySQL storage formats (like the default MyISAM) don't have this ability.

By "simple" I didn't mean simplistic, I meant like K.I.S.S.! :)

I'm somewhat more interested in being able to just do a backup on demand (I backup my important files to AWS daily) and grab a copy of that backup for my own archives (i.e. burn a dvd set).

The first is taken care of by the user-selectable backup, but will the second be available? I.E. Will I ever be able to grab a disk image?

…Chris

@phvt:

@caker:

I beg to differ :). This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.
My understanding is that LVM snapshots can be made of running InnoDB tables and restored successfully. The InnoDB tables will be in "crashed" state, but the data will recover automatically.
Yeah, but there's a potential snag when involving a virtualization layer:

We're issuing the LVM snapshot on the host – not from within your Linode's kernel. Performing a snapshot flushes dirty buffers, but since that's being done on the host your Linode kernel is unaware that this is happening, so it still has dirty buffers that were never written to disk.

This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file…

-Chris

@xoff00:

The first is taken care of by the user-selectable backup, but will the second be available? I.E. Will I ever be able to grab a disk image
Restoring restores to disk images, which you can then mount and pull down on your own with whatever (rsync, scp, tarball, etc).

-Chris

@zengei:

Just making it clear: ext4 disk images are not currently supported, correct?
They will be very soon – maybe a few days.

-Chris

@caker:

@zengei:

Just making it clear: ext4 disk images are not currently supported, correct?
They will be very soon – maybe a few days.

-Chris
Great, look forward to it.

@caker:

Yeah, but there's a potential snag when involving a virtualization layer:

We're issuing the LVM snapshot on the host – not from within your Linode's kernel. Performing a snapshot flushes dirty buffers, but since that's being done on the host your Linode kernel is unaware that this is happening, so it still has dirty buffers that were never written to disk.

This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file…

Thanks–this is the detailed explanation I was looking for--how these snapshots are different from direct LVM snapshots, and why a db dump is very much necessary.

I would definitely like to participate in the beta whenever it is available in Dallas.

I will definitely purchase this option whenever it becomes available!!!

Jeff

Just tried to backup my 360. I'm only using 1.9GB, and the backup took 5 hours!

snapshot 2009-04-03 21:07:16   2009-04-03 21:10:42   3 minutes, 26 seconds        successful
snapshot 2009-04-03 14:21:03   2009-04-03 19:33:48   5 hours, 12 minutes, 45 seconds       failed

@saman007uk:

Just tried to backup my 360. I'm only using 1.9GB, and the backup took 5 hours!
I'm not surprised at the failures – I've been taking things up and down all day, making improvements for speed.

-Chris

@saman007uk:

Just tried to backup my 360. I'm only using 1.9GB, and the backup took 5 hours!
Didn't you see the note that said the backup server is connected via dial-up from caker's basement? ;)

Just to be clear: we don't have to report failures? I would guess you get notified of them and details now its in beta….

I say his because my on-demand backup failed; you were probably tweaking :D

Just checking, but if i use paritions on a drive.. or more to the point put a "drive" into an LVM.. and then parition on top of that im going to assume im not going to be able to use this backup option correct?

If that is the case, would you consider pricing by size rather then node type? Because if i could only backup say my root non lvm image, im not really able to back up all that much data..

@Internat:

Just checking, but if i use paritions on a drive.. or more to the point put a "drive" into an LVM.. and then parition on top of that im going to assume im not going to be able to use this backup option correct?

All I could think up for this situation so far is to make a snapshot, mount a "normal" disk image and then rsync the snapshot to that, prior to the backup window. But now you've made your own backup (using more expensive disk space)…

@caker:

This may be moot because of journaling (both at the filesystem and the db level), but, speaking for myself, I'd dump my DBs to a file… It would be nifty to expose this functionality to the API. :)

@caker:

@phvt:

It doesn't sound like it's an LVM-based snapshot, and instead is running rdiff/rsync or something simpler. The original post says we'll need to dump databases to a file prior to the backup window.
I beg to differ :). This is a very sophisticated system which does involve LVM snapshots on the host. But what's the difference? Live database files can't be snapshotted, or copied, or rsync consistently, anyway.

@saman007uk:

Will this take an actual "snapshot" or will we need to stop MySQL, etc. in order to prevent file corruption.
Doing a snapshot does NOT always mean you're doing to get your mysql DB files in a consistent state. It can depend on which type of table engine you're using, for instance…

So dump your DBs to a file before the backup window just like you would with any other backup configuration.

-Chris File databases like sqlite3 needs special care (dump before backup) like mysql, or they just get backuped without problem?

Well, I have a suggestions if it isn't already implemented, the ability to mount backup images so you can pull single file backups off of them.

@Andrea:

File databases like sqlite3 needs special care (dump before backup) like mysql, or they just get backuped without problem?

Just put a short script into your crontab to dump your DB every day into an sql file or something just before the backup. This is all special care you need!

@RossH:

Well, I have a suggestions if it isn't already implemented, the ability to mount backup images so you can pull single file backups off of them.

I second the above idea. This would introduce amazing flexibility into the system and possibly prevent unnecessary downtime. Is there anything like this yet?

what's the difference between this service,

and "duplicate image" in my dashboard?

With duplicate image I can create a really perfect backup,

so how this service can improve the old one?

1. It's cheaper than your own "online" disk space, 2. it doesn't require downtime (like duplicate image does), 3. and it keeps multiple copies (see #1) that are safely stored on a different server.

:)

Any news?

Yeah, what started as "down for the weekend" became "offline for over a month". Any idea when will the service be available again? Especially after the Dallas incident (which resulted in total data loss) backups became more important.

> Especially after the Dallas incident (which resulted in total data loss) backups became more important.

What do you mean "more important"? I find backups have always been important. You should have them, make them, test them.

Did you not have backups of your stuff?

I don't know if you were backing your data up or not, but here is my opinion on the subject:

To quote myself (http://www.linode.com/forums/viewtopic. … 1423#21423">http://www.linode.com/forums/viewtopic.php?p=21423#21423)
> This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.

Even if/when Linode gets backups working, you are still responsible for your data. I'll subscribe to their backup plan (depending on price point) for convenience, but I'll also be doing my own backups independently or downloading my backups from Linode (if that's possible) and testing them to make sure the data is there and uncorrupted. Which is why I want to get on the beta when it becomes available again so I can see what it's all about and how it works.

In my opinion, when scaling for a linode you should also plan for space to hold a backup.

When I started with linode, a linode64 came with 4.5Gb of disk (if you paid yearly and so got 1.5Gb extra free). Now the minimum comes with 16Gb disk (or 24Gb if grandfathered in with old bonus plan). I've only allocated 60% of the disk space I have, so far!

This means I have plenty of backup space. In this I do a weekly level 0, daily incrementals. And these get rsync'd daily to my home machine.

So if things totally utterly exploded I'd need to recreate disk profiles, boot into finix, scp up the backup files, restore data. Of that the scp would probably be slowest since my upload speed from home is only 5Mbit/s and my level 0 gzip'd files come in around 2Gbytes in total.

Of course, I also run an essentially mirrored configuration at Panix, so I'd just have to modify DNS to point there while I was rebuilding.

Besides of the fact that I'm still eagerly waiting for the backup service to come back, I always used only the half of my disk space for images. The other half is still handy for emergency backup images. Of course this no way replaces any of the decent backup solutions but it came handy for me once.

Hi Linode, How can I participate in beta? And when this service come to commercial?

I too would like to subscribe to this service.

@caker:

Open a ticket under that Linode and request that we enable backups.

@caker:

What about the other facilities?

Atlanta and Dallas sometime later this month, and Fremont early May.

-Chris
Hello,

Are these available in Fremont now, I'll open a ticket if that's the case.

I'm in Fremont and on the beta.

The beta is currently available in every datacenter except for London.

Is (beta) backup feature in plan for London datacenter?

@kornrunner:

Is (beta) backup feature in plan for London datacenter?

From what I've heard: Yes,, the next version of the backup system will be available in London as well. There is no ETA as to when that will happen.