What's a decent way to backup a linode

Our linode has a three-tiered backup system:

Tier 1: Linode's backup service

This is the first stop, because it can be used to recover a damaged/failed/etc linode extremely quickly. A new linode can be deployed from the backup in a matter of minutes.

Tier 2: Rsync'd backups to an offsite ZFS fileserver doing incremental snapshots

If Linode's backups aren't working, aren't available, are corrupt, etc, we fall back to this tier. Every night, I have my ZFS file server (which currently has two RAIDZ2 arrays of 7x4TB and 8x2TB drives, which are roughly analogous to RAID6). A script runs on the file server which connects to the linode via SSH, issues a MySQL database dump command, compresses it, and then performs an rsync of the entire filesystem, with many system and log files and other frequently changing files excluded. That's because this is intended to be a data backup, not a system backup: that is to say that we can use it to rebuild our system quickly, get our sites back up and running, not to be an exact copy.

If we need to restore from this backup, we'd deploy a new linode (or a new server elsewhere), set up the same distro as we currently use, copy the configs from the backup, and then copy the site and database data. It's not nearly as fast a process as on-site backups, but it gets us up and running quickly.

The ZFS server (which, because LZ4 is extremely fast, runs a fully compressed 44TB filesystem) does nightly ZFS snapshots right after it finishes the rsync backups (it actually does the process on a few servers, not just our linode). I periodically go in and manually trim the snapshots (ZFS lets you delete ranges of snapshots). My policy is to keep nightly snapshots for the month-in-progress and the previous month, and then monthly snapshots after that.

Tier 3: Personal backups

This is sort of a quasi-tier, but it's worth including because we had to use it before we had a proper backup plan. This involves rebuilding our server from database dumps that our developers periodically make for their own use, as well as copies of the code they have on their home machines.

Before we were at Linode (or had a backup policy), our old VPS provider decided to move all his hardware to a new datacenter, without notifying customers. Downtime was, if memory serves, two or three weeks, with zero communication to customers for the first week or something. It was the sort of thing only an idiot would do, so we immediately began migrating to a more reputable host. It was a painful experience, and we did suffer some data loss from the database dumps being slightly out of date (including customer registration/payments, forcing us to use PayPal's transaction history to figure out which customers we had to contact and ask to re-register), and it caused us to put a proper backup plan in place, but it did work.

For our static files, we use rdiff-backup to sync everything to an off-site server. rdiff-backup is simpler than rsync and has many more features.

For MySQL data, we use Linode's backup service as our first tier. For the second tier, we have a replication slave offsite that's snapshotted once a day.

As for what to backup, we use Puppet to provision and manage our servers, so we don't bother backing anything up that can't be recreated with Puppet. If you use any sort of configuration management system, you should be able to easily determine what needs to be backed up by comparing the filesystem for a newly-provisioned box with one that's been in production for awhile.

I reject the assertion that rsync is complex or difficult, rdiff-backup just does a bunch of stuff by default that rsync would need a switch for. So use an alias or something and you're back to a "simple" command-line. Furthermore, rdiff-backup's extra features aren't required if you're doing snapshotting at the filesystem level. In fact, filesystem snapshots are way easier to work with than revere-diff files that rdiff-backup produces, because filesystem snapshots can be used without any special tools (like having to execute the reverse diff). For btrfs, you just mount the snapshot. For ZFS, it's even easier, you just cd into the snapshot hidden directory, and all the files in the folder are there just like they were when the snapshot was taken.

@Guspaz:

I reject the assertion that rsync is complex or difficult, rdiff-backup just does a bunch of stuff by default that rsync would need a switch for.

rsync is simple if you're an experienced Linux user, but it's pretty clear saudor is a newbie. For new users, rdiff-backup's website is much friendlier than the rsync man page.

@Guspaz:

Furthermore, rdiff-backup's extra features aren't required if you're doing snapshotting at the filesystem level

True. The reason I didn't go that route is because the remote server I'm synching to doesn't use a filesystem that supports snapshots, and I didn't want to go through the effort of repartitioning it.

I figured it out using rsync. Turns out i was doing something stupid (early morning) +my ssh keys were being weird which prevented the connection from even happening. refreshed both and all is good now.

I just wrote a .sh script to rsync my /var /home /srv/www /etc /usr onto the internal hard drive on my computer, compiled and threw it into cron.

i just pointed Apple's Time Machine to that folder on my HD which round robins to two external disks so that should be ample i think - at least in terms of "daily, weekly, monthly snapshots"

For extra protection, i'll probably just symlink the most important parts of that backup (i.e. MySQL DBs) to an encrypted TrueCrypt Volume on dropbox.

I have all my code/php scripts in that volume so in the worst case scenario, i can just rebuild everything but that's tedious hence looking for a better/less bandwidth/faster solution.

As for Linode.. i used that backup for a few months but since it works on the entire image, it didn't really fit my needs as often times. i want to restore a single messed up file and not have to restore the entire server.

@saudor:

And more importantly.. what needs to be backed up besides the home directory and the /srv/www directories of course.
If you can afford the disk space, this is the option I use…

1) Create a new /BACKUP partition

2) Run daily 'dump' incrementals (Sunday == level 0, Monday == 1, Tuesday == 2…) of all the other partitions to /BACKUP

3) Delete old backups (eg over 14 days old)

4) rsync'd the /BACKUP directory offsite (in my case my home machine initiates the rsync)

If you have databases running (eg mysql) then do a DBdump to the /BACKUP partition as well.

Yes, this wastes disk space, but it's a very traditional way of doing Unix backups; it's worked well for me for over 20 years and it's a much much older design (although in those days it went to tape) :-)

I also use this for my home machines; at one point a physical machine died (while I was in an airport!!) and I managed to build a VM and restore the backup via SSH and left the restore running when I got on the plane; by the time I got the hotel it'd finished :-)

That's actually a pretty neat idea.

Im on the cheap linode so not too good on space… though I'm using about 12 GB out of the 48 GB. That being said, i can probably get away with weekly backups on the file system as that does not change too much. The database however changes constantly so i have daily backups stored on the server for the past 14 days and automatically rotates out.

Every week i go in and download those archives to my external hard drives with the latest uploaded to dropbox. I really should slip this into my script so it does it automatically heh

+1 on the VM approach. Currently, i keep a bootable image of my SSD partition. Plug it into any computer and all my apps are there - in case the physical machine dies.

As for data, most of my files on Dropbox (which of course is backed up to multiple HD). That way, it syncs seamlessly across both my Mac and PC. Ive done work right from my iPad and iPhone too and by the time i get home, everything is waiting for me on the computers. Pretty neat.

"What's a decent way to backup a linode"

I usually use Post-It Notes.

YMMV

In addition to the Linode backups, I use a CDP backup service which runs on software by Idera (formerly known as R1Soft). It creates a block-by-block backup of the entire server every hour (it only transmits the deltas, so there's not too much data to be sent) and you can customize how long these incremental backups are stored. This effectively creates a bunch of snapshots for every file on your system (it also has a special plugin for MySQL databases), which you can restore in a few clicks, just as you would restore revisions in Apple Time Machine backups or Dropbox.

I'm using a third-party provider which runs the backups server in Los Angeles (my Linode is in Newark). I'd be happy to recommend them, although I'm not sure what the policy is here for mentioning other companies. You can PM me if you want more info.

You asked for a decent way, this is what I do:

#!/bin/sh

#[Note: This is a FULL system backup script and requires root. If you
# only want to backup your user files then tailor the script.]
# Use "sudo crontab -e" to set up a cron job to run it.
#
#[Note: --delete will remove target files and dirs that no longer exist in
# the source, you may or may not want this sync'ing.]
#
#[Note: The first backup will take a while, to add the files to the
# target, after that it should only take a matter of minutes.]
#
#[Note: rsync must be installed on the source and the target.]
#

BINPRE="rsync -r -t -p -o -g -v -l -D --delete"
SSH="-e ssh -p 22"
BINPOST="<target_user>@<target_host_ip>:/<target_backup_dir>"
EXCLUDES="--exclude=/mnt --exclude=/tmp --exclude=/proc --exclude=/dev "
EXCLUDES=$EXCLUDES"--exclude=/sys --exclude=/var/run --exclude=/srv "
EXCLUDES=$EXCLUDES"--exclude=/media "

date >> /root/start

$BINPRE "$SSH" / $EXCLUDES $BINPOST

date >> /root/stop</target_backup_dir></target_host_ip></target_user> 

Edit: This is how I backup my databases daily using cron jobs:

MySQL

0 1 * * * /usr/bin/mysqldump --all-databases --events | gzip > /root/databasebackups-mysql/database_"`date | tr \" \" \"-\"`".sql.gz

PostgreSQL

0 2 * * * /usr/bin/pg_dumpall -h localhost -U postgres | gzip > /root/databasebackups-postgres/database_"`date | tr \" \" \"-\"`".sql.gz

rsycn is a good tool. You should use it along with shell scripts to back up the databases and have the backup files exported with the rsync process. Then compress the rsync backup with a filename that corresponds to the date.

When backing up databases, it is essential to do a database dump, as the files that are associated with the databases will not work on other systems.

https://www.opsmate.com looks like a decent option too. Still in beta though.