Is it just me, or is the Linode backup service about useless

I've never used Linode's backup service and never intend to. It strikes me Linode's backup service is there for convenience and not because it's the right way to do things. Good backups have to be off site, if Linode got cracked or went bust these backups are just as vulnerable as the live images. If the data center your Linode is at goes bust overnight, gets raided by the police, losses its Internet access due to a peering dispute, a careless road digger, or whatever, you lose the live systems and the backups at the same time.

The right way to do backups is remotely and at a schedule appropriate to the data. I have BackupPC running at home doing nightly backups. If every Linode site got nuked from orbit I could take any nightly backup and restore it to a dedicated or virtual machine anywhere.

I use exactly the same setup for the company Linodes I manage, regular backups to the company backup server.

I use the Linode backup service together with other forms of backup. This way, I have an easy way to restore (Linode backup) along with always having my data - even if the whole dc go down.

When you want to enable backups on a Linode, the following is displayed on the confirmation page:

> The Linode Backup System is designed to be an easy to use, reliable and redundant on-site backup solution for your Linode. It performs backups without causing any interruption of your running system. It provides 4 backup slots. Three of the slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. The fourth backup slot is a user-initiated snapshot and remains in place until another user-initiated snapshot is taken.

The library article about the backup service explains it like:

> Daily backup: Automatically initiated daily within the backup window you select. Less than 24 hours old.

Current week's backup: Automatically initiated weekly within the backup window, on the day you select. Less than 7 days old.

Last week's backup: Automatically initiated weekly within the backup window, on the day you select. Less than 8-14 days old.

Manual Snapshot: A user-initiated snapshot that stays the same until another snapshot is initiated.

The daily backup you want will never rotate into the weekly slot. It was destroyed when a new daily backup was taken. Only the daily backup taken on the weekly backup day will rotate into the weekly slot.

As an example, I have three automatic backups right now: May 5th, April 28th, and April 21st. Sunday is my weekly backup day, and May 5th (today!) is a Sunday. Yesterday, my daily backup was May 4th and my weekly backups were April 28th and April 21st. Tomorrow, my daily backup will be May 6th, my latest weekly backup will be May 5th, and my older weekly backup will be April 28th. The May 4th backup does not exist any more as of May 5th, but the May 5th backup will stick around until May 26th.

Myself, I generally use the Linode backup service for fast/easy full restores in the event of calamity (hardware failure, wrong buttons, etc). The huge advantage of it is the ability to do a full restore and be back online very quickly without a lot of fuss. For partial restores, or in the event Linode itself is AFU, I also use duplicity to back up stuff to S3. The full restoration process with duplicity is relatively terrible, but it's much nicer for partial and/or point-in-time restores.

tl;dr: Linode backup service is awesome at what it does, not so awesome at what it isn't designed to do. Things like duplicity are awesome at what Linode backup service isn't designed to do, but not so awesome at what Linode backup service does. If your data are important to you, use both.

Well, the consensus seems to be that it's a very basic no-frills service, and that I shouldn't expect much. In fact, I would venture to say it's one of those "you better hope you never need it, because it sucks" situations.

Thanks for all the comments online and offline. I think I'll just cancel it and move on to something else.

Does anyone have any good backup scripts they want to share? Debian 6 on my boxes.

thanks,

bruce

It doesn't suck, it does just what it says on the tin… It's an important part of an overall backup strategy, it's not a backup strategy unto itself.

For our part, we use the Linode backups as our first tier on-site backups. In case of emergency, our first step is to restore from our Linode backups, because that's the fastest approach; we can spin up a new Linode from our backup in a matter of minutes.

If that fails for whatever reason (Linode loses the whole datacenter, backup is corrupt, whatever), we go to our next tier. That involves nightly off-site incremental backup of non-system files. Every night, my home fileserver remotely initiates a MySQL dump and then rsyncs our linode to a local directory and takes a ZFS snapshot. If we need it, we deploy a new linode from the distro template and copy the data back from my fileserver. Takes longer, both because it involves some manual rebuilding and it's from a residential broadband connection (albeit one with 10 Mbps of upstream), but it's not that much work.

Our third tier, such as it is and what little there is of it, involves gathering up backups of individual chunks of data from various developer's personal machines to rebuild our infrastructure piece by piece. This sucks, and is a long and painful process. In fact, our having to execute it (when a previous VPS hosting provider took our servers on a literal road trip without informing us) and the pain it caused us (particularly since we had to do the rebuilding from the back of a van in the middle of a drive from Montreal to Toronto, a 6 hour trip) is what prompted us to put the first two tiers in place. There are now two far less painful procedures in place that we can use before we ever have to consider rebuilding from scratch like that again.

I also use the Linode backups as a first tier of a multi-tier solution on my main production nodes. It's definitely not the right approach if you want single file restorations or different backup time cycles than its fixed slots, but it's a nice safety net for my more critical nodes as a primary bare-metal (well, bare-VPS) restore option.

I've also found having backups enabled on a production node to be quite convenient for development/test purposes, since it essentially enables cloning the node without having to shut it down, either by using the most recent backup, or by making a snapshot right at the point you need to clone. To be honest, to date that's what all of my restorations through the backup system have been for.

Outside of Linode, for my part my primary backups are through a centralized bacula system (that I use for a wide variety of machines, both Linode and non-Linode) with backups stored on a local RAID box (the bacula director and storage agent runs right on the box). It's got a learning curve, but gives me complete centralized control over the individual backup cycles - individual machines get an agent installed. In my case I operate with fairly classic incremental, differential, and full backup cycles. This is almost always what I turn to for restoration, since in most cases it's a single file or small set of files I'm interested in, as opposed to a bare metal restore. It could also restore all local files on top of a new Linode template installation if needed.

– David

@bbergman:

Does anyone have any good backup scripts they want to share? Debian 6 on my boxes.

BackupPC works wonderfully but if it's overkill for you then rsync with a small shell wrapper kicked off from cron might do what you want.

I use BackupPC, I'm very happy with it.

@bbergman:

Does anyone have any good backup scripts they want to share? Debian 6 on my boxes.

I'm currently using a headless install of crashplan to backup to my always on network storage at home.

What's wrong with rdiff-backup? :)

No the backup service works exactly as documented and I have used it successfully.

I have encrypted backup of databases and filesystems in cloud storage and offline if I need to recover specific files or the whole of Linode goes up in flames. But it would take forever to restore an entire functioning server from them. When Fremont had one of its frequent power outages some time ago it corrupted one of my instances. I did a restore from backup to a different server and there was minimal downtime. IMO the Linode backups are for quick disaster recovery if a Linode instance shits itself and they do that really well. If you need something more fine grained you go to your other backups because you don't keep all your eggs in one Linode basket do you?

My primary backup strategy for the last many years has been rsync with the –link-dest option to create hard link-based snapshots. rsnapshot is a good wrapper around this.

But I'm hoping to make all that obsolete with a server backup service I've been working on. It'll have the simplicity of the Linode backup service but without the limitations. You'll be able to do a "bare metal" restore of an entire snapshot, or simply restore individual files. I should have a beta available in a couple months. If anyone wants to learn more or is interested in using the beta when it's available, send me an email or a PM. I'd also love to hear what features you'd want to see in an ideal server backup service. :-)

3 Tiers.

o Linode Backup

o Daily tarball backup of specific important data areas (web trees, configure files etc).

o Duplicity Backup to my home office server.

Tarball keeps 3 days rotating. Good for when you fat finger something and need a quick restore of it or a number of files.

Duplicity in case I need to start over elsewhere and the DC is down.

Linode for restoring to another Linode or a major crash.

Each has their own strengths.

After reading this thread I think Linode backup service is overpriced. For that price I want to get full-featured backup, not only "better than nothing".

Update: I have finally found the (almost) perfect backup service, and it couldn't be easier to use with Linode (and another service I use that has droplets). It's here: http://www.opsmate.com

The install was super easy, and it immediately did a full backup of my Linode, and started doing differentials every night. I love that I can browse the web and view the snapshots, decide which to download (restore) and so on. There's also a very cool interactive FTP-like command line tool that I can use in the shell on my linode.

Although I haven't tried it yet, I can also mount the backups as a FUSE filesystem and do more advanced things such as diff'ing files and such.

Although it's still in beta right now, it's solid enough that I'm finally dropping linode's backup service. This is FAR better, and more in line with my expectations for being able to restore what's needed, when needed, as needed. You should check it out if you're still searching for a decent backup service.

Doing a happy dance… :-)

Thanks,

Bruce

Looks interesting, shame it's not open source. Have you done a restore to verify it works as expected?

I haven't done a full server restore yet, but yes, I've done several spot check restores, including full directories, and everything worked like a charm.

@bbergman:

Update: I have finally found the (almost) perfect backup service, and it couldn't be easier to use with Linode (and another service I use that has droplets). It's here: http://www.opsmate.com
Thanks for the link. I signed up for beta, looks really promising.

@bbergman:

Update: I have finally found the (almost) perfect backup service, and it couldn't be easier to use with Linode (and another service I use that has droplets). It's here: http://www.opsmate.com

Thanks for sharing! That looks really promising.

Hey everyone, Opsmate founder here. I'd be happy to answer any questions you may have about the service.

I take Linode's backup service for what it is. They say that RAID1 (or 10 for that matter) is about availability, not backup, right? That's exactly why I'm paying Linode a (very humble) extra so that I can take a snapshot whenever I mess around with a production server. If that [apt-get dist-upgrade] goes wrong, I can redeploy my Linode in a matter of minutes and begin the brainstorming process about where exactly it went wrong. Availability, baby! (Actually, I'd clone the Node from my snapshot for what, couple cents, stage the upgrade there and then upgrade on production).

That said, I do not rely on Linode's backup service as my sole backup. That'd be silly, come on. Eggs and baskets, remember? If worst comes to worst (Linode backup went FUBAR, Linode itself went belly up, 3-letter agency pulled an entire rack including the backup server, you name it…) I rdiff to one local and one remote server anyway. Granted, this will take time to be up and running again, so there goes the availability but my data is safe.

As for Opsmate, it does look great really, but I find the pricing rather steep. I do realize there's a lot of compression and optimization going on, but that really only applies to the OS and log files. The actual data, on my Linode at least, is oh so very not compressible (RAW photo files anyone?). My Linode 4096 (196 GB storage) runs me $80 plus $20 backup. Backing it up to Opsmate would cost me another $75 for 200 GB storage. (Like I said, my data doesn't compress well, so the 100 GB plan won't do). Well, frankly, for $75 I'd rather just spin up another Linode in another datacenter and rdiff.

Bottom line, I love Linode, they've been extremely reliable in my experience and I appreciate their backup feature for its convenience. I will not exclusively rely on it for reasons mentioned earlier, so I do keep my 2nd and 3rd backup and backup of the backup elsewhere, but I won't pay another $75/mo for it.

It isn't really difficult to write a script to back up to S3 on your own. Here's an example:

#!/bin/bash

BACKUP_DIR="/backups"
MYSQL_FILENAME="$BACKUP_DIR/prod-$(date +%m%d%Y).sql"
DOCROOT_FILENAME="$BACKUP_DIR/prod-$(date +%m%d%Y).tar.gz"
ETC_FILENAME="$BACKUP_DIR/etc--$(date +%m%d%Y).tar.gz"
BACKUP_HISTORY="365 days"

echo "Creating backups..."

mysqldump -u username --password=password database > $MYSQL_FILENAME
gzip $MYSQL_FILENAME

pushd /opt
tar cfz $DOCROOT_FILENAME www/prod/ scripts/
popd
pushd /etc
tar cfz $ETC_FILENAME php* nginx* sphinx* my.cnf
popd

s3cmd put ${MYSQL_FILENAME}.gz s3://your-s3-path/ && rm -f ${MYSQL_FILENAME}.gz
s3cmd put $DOCROOT_FILENAME s3://your-s3-path/ && rm -f $DOCROOT_FILENAME
s3cmd put $ETC_FILENAME s3://your-s3-path/ && rm -f $ETC_FILENAME

echo "... done."
echo
echo "Cleaning up old backups, deleting files older than $BACKUP_HISTORY..."

s3cmd ls s3://your-s3-path/ | while read -r line;  
do
   createDate=`echo $line|awk {'print $1" "$2'}`
   createDate=`date -d"$createDate" +%s`
   olderThan=`date -d"-$BACKUP_HISTORY" +%s`

   if [[ $createDate -lt $olderThan ]]
   then
      fileName=`echo $line|awk {'print $4'}`

      if [[ $fileName != "" ]]
      then
         s3cmd del "$fileName"
      fi
   fi
done

echo "... done."

robert

Or rsync to another host:

#!/bin/sh

#[Note: This is a FULL system backup script and requires root. If you
# only want to backup your user files then tailor the script.]
# Use "sudo crontab -e" to set up a cron job to run it.
#
#[Note: --delete will remove target files and dirs that no longer exist in
# the source, you may or may not want this sync'ing.]
#
#[Note: The first backup will take a while, to add the files to the
# target, after that it should only take a matter of minutes.]
#
#[Note: rsync must be installed on the source and the target.]
#

BINPRE="rsync -r -t -p -o -g -v -l -D --delete"
SSH="-e ssh -p 22"
BINPOST="<target_user>@<target_host_ip>:/<target_backup_dir>"
EXCLUDES="--exclude=/mnt --exclude=/tmp --exclude=/proc --exclude=/dev "
EXCLUDES=$EXCLUDES"--exclude=/sys --exclude=/var/run --exclude=/srv "
EXCLUDES=$EXCLUDES"--exclude=/media "

date >> /root/start

$BINPRE "$SSH" / $EXCLUDES $BINPOST

date >> /root/stop</target_backup_dir></target_host_ip></target_user> 

The problem with rsyncing to another host is that you only have one copy. If something gets corrupted, hacked, or whatever today, and the backup/rsync script runs tonight, you've lost any chance to recover things. With my script (or any other 'real' backup system), you have 365 (or however many you choose) days of backups to choose from.

robert

@robertcope:

The problem with rsyncing to another host is that you only have one copy. If something gets corrupted, hacked, or whatever today, and the backup/rsync script runs tonight, you've lost any chance to recover things. With my script (or any other 'real' backup system), you have 365 (or however many you choose) days of backups to choose from.

robert

I provided a basic script so anyone can figure it out and get it running. I run my script on the client (Linode here) nightly.

On my target server I run a weekly cron too, a weekly location on the target, from daily target to weekly target locations on the target, I do monthly too. It's not perfect, yours is better in that you can choose any day of the year so long as you're willing to pay for S3 storage.

Mine works with any target system, any Linux, Mac or even Windows (there are ssh and rsync solutions on Windows) where you can run:

1) ssh

2) rsync

That's all you need. The target is where you can break out weekly, monthly, bi-daily, whatever periods you want, with cron, as long as your target has the disk space available.

edit: rsync saves immensely on disk wear and tear too.

If you want to save multiple versions, I suggest using rsync for the current updated backups, and then run rsnapshot {daily, weekly, monthly, etc.} for the multiple versions (using hard links – very fast and saves space).

Lester

Thanks for pointer to opsmate.com - I'm going to check them out.

@bjl:

@bbergman:

Does anyone have any good backup scripts they want to share? Debian 6 on my boxes.

I'm currently using a headless install of crashplan to backup to my always on network storage at home.

Thank you for mentioning this. I've been using Crashplan on my home computers for a while now, I don't know why I never thought to use it on my server. But it's done now :)

This post has been up for a bit, so I figured it was time for an update by a Linode employee (myself).

Great! I go to the list of backups, and peruse the nightly backups. There's the one I wanted, three days ago.

I read the docs on how to restore, but can't find a way to restore this particular backup set. I ask support, and they tell me I can only restore from one of the four pre-determined backup lots (last night, last week, two weeks ago, or my last manual backup).

In our Classic Manager (now retired), you were able to see a log of all of the Backups that ran over a given period of time. We had some customers express their confusion that even though that log was there for informational purposes, it could be interpreted that those listed were all Backups that could be restored. To reduce confusion, we have removed that feature in the Cloud Manager so now all you see are the four Backup slots from which you can restore.

It's been gone over in this thread already, but it's worth an additional mention that the four Backup slots are as follows:

• Daily backup Automatically initiated daily within the backup window you select. Less than 24 hours old.
• Current week’s backup: Automatically initiated weekly within the backup window, on the day you select. Less than 7 days old.
• Last week’s backup: Automatically initiated weekly within the backup window, on the day you select. Between 8 and 14 days old.
• Manual Snapshot: A user-initiated snapshot that stays the same until another snapshot is initiated.

The daily and weekly backups are automatically erased when a new backup is performed. The Linode Backup Service does not keep automated backups older than 14 days.

The above was taken from our guide on how Linode Backups work

Just FYI, anyone, our Linode's automatic backup failed six days ago and we haven't had a backup since. Can't even take a manual snapshot. Just got a ticket update from them (6 days later) saying this is because of "normal maintenance" (emphasis added). I'm hoping that was just poorly thought-out / badly written because no "normal" process should cause a problem like this.

(I am doing an rsync to a local-to-me file server, but that's beside the point.)

@AASV I wanted to respond as I'm a Backup Admin, and I think you could benefit from some more insight into what's currently happening with your backups. As our status page states, we are currently performing some maintenance on storage boxes in our Dallas data center. The maintenance that is occurring is affecting a smaller subset of storage boxes, though the nature of boxes like this, it includes a large amount of data and customers' backups on each box. This maintenance is independent of any other work we're performing on hosts where your Linode is on.

We're working as quickly as we can to lighten the load on the boxes that need maintenance. Because of the amount of data and files included, some customers may not see relief for a couple more days. We're moving backups in batches, though if you open or respond to a Support ticket, we'll do our best to get your backup moving asap.

Thank you, @watrick, for the additional information.