Backing up Linode files with s3sync

I got mine working using s3 + fuse.

http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Works like a champ on Ubuntu Hardy!

Okay, the fuse plugin is slow, so I rewrote myself some backup foo to work with s3.

I used the s3cmd packaged with ubuntu hardy. This script assumes you already have s3cmd installed and configured properly.

A couple of TODOs:

• Incremental backup

• Auto-pruning

I'm using this on my linode to great success. Just note you have to have enough free space for "scratch" to compress each chunk you're backing up.

I just print everything to stdout, and since I run the script in cron, it gets emailed to me.

#!/bin/bash
NOW=`date +%d-%m-%Y`
echo "====Beginning Backup for $NOW===="
echo ""
echo "Creating bucket viagraoldos-$NOW"
s3cmd mb s3://viagraoldos-$NOW

for i in home etc root; do
        echo "==Beginning Backup of /$i=="
        echo ""
        echo "Making a tarball of /$i..."
        tar cfj /tmp/$i.tar.bz2 /$i
        echo "Done!"
        echo ""
        echo "Uploading tarball of /$i to s3..."
        s3cmd put /tmp/$i.tar.bz2 s3://viagraoldos-$NOW/
        echo "Done!"
        echo ""
        echo "Deleting local copy of tarball of /$i..."
        rm -r /tmp/$i.tar.bz2
        echo "Done!"
        echo ""
        echo "==Backup of /$i completed=="
        echo ""
done

for i in backups www; do
        echo "==Beginning Backup of /var/$i=="
        echo ""
        echo "Making a tarball of /var/$i..."
        tar cfj /tmp/var-$i.tar.bz2 /var/$i
        echo "Done!"
        echo ""
        echo "Uploading tarball of /var/$i to s3..."
        s3cmd put /tmp/var-$i.tar.bz2 s3://viagraoldos-$NOW/
        echo "Done!"
        echo ""
        echo "Deleting local copy of tarball of /var/$i..."
        rm -r /tmp/var-$i.tar.bz2
        echo "Done!"
        echo ""
        echo "==Backup of /var/$i completed=="
        echo ""
done

echo "====Backup completed. Objects currently in s3 listed below.===="
s3cmd la

````
s3cmd mb s3://viagraoldos-$NOW

````

You should use "s3cmd –ssl" or your AWS secret key is sent in plain text.

I have a script that uses s3sync to sync certain configured directories to a S3 bucket, then uses s3cmd to upload a tar/bz2 of other directories. It uses the --exclude features to not sync certain directories and it also auto-prunes the tar/bz2 files that are older than some number of days.

If you are interested, I will upload it. I just need to spend 5 minutes removing ugly code and important info like my access ID and secret key :)

--Eric

@ryantate:

Note: If you use the verbatim SSL cert provided in the README (and referenced, I believe, in the tutorial), you need to name it "f73e89fd.0", which is not mentioned anywhere but the main Amazon website thread on s3sync:

http://developer.amazonwebservices.com/ … 5&tstart=0">http://developer.amazonwebservices.com/connect/thread.jspa?threadID=11975&tstart=0

You can do this to generate appropriate .0 for any cert by doing, via sh/bash:

$ HASH=`openssl x509 -noout -hash -in ${CERT}`
$ ln -s ${CERT} ${HASH}.0

Where CERT is filename of the SSL certificate itself.

I built s3fs http://code.google.com/p/s3fs/wiki/FuseOverAmazon and used duplicity http://duplicity.nongnu.org for simple backups.

i wrote a small script that mounts my s3 account and runs duplicity to make the backup, and then unmounts.

simple and quick, but for larger systems it certainly won't scale.

• j

I use s3 & fuse with "simplebackup" (http://sourceforge.net/projects/migas-sbackup) - works perfectly.

When mounting the s3 bucket i got error messages but following the instructions in this thread http://www.linode.com/forums/viewtopic.php?t=3114 helped.

simplebackup allows for incremental and differential backups - a great way to save on space & bandwidth.

Hello

You don't need to use Fuse with duplicity and S3.

Duplicity have native support to S3.

Your you can do the backups directly on S3 without mounting any filesystem.

You must install from source latest duplicity version with Boto python library.

Take a look:

http://www.linode.com/forums/viewtopic.php?p=14974

regards

roberto

Please send me your feedback : hareem.haque@gmail.com

Semi Automated Uploads to S3 bucket. Data offloading

#!/bin/sh
#This script first asks for a users input. And then it will sync content of a specific directory
#with a S3 bucket.

#Specify all the data related to your AWS Account.
#Please remove the "" prior to entering your credentials.
# PLEASE NOTE THAT I HAVE NEVER TESTED THE SCRIPT WITH AN EU BUCKET. 

export AWS_ACCESS_KEY_ID="YOUR ACCESS KEY HERE"
export AWS_SECRET_ACCESS_KEY="YOUR SECRET KEY HERE"

#Enter all required info below. Make sure to double check everything that you specify.

echo -n "What directory would you like to offload to S3 ? "
read bkupdir

echo -n "Please specify your bucket ?"
read Bucket

echo -n "Specify your prefix? "
read prefix

echo -n "Specify log name for this process. Please specify a time input "
read mlog

echo  -n "Specify log dir path. (plese specify a path to your log directory - Please inclde a /): "
read ldir

#Script will ask you all the required questions.
#Please make sure that you specify the path to your s3sync folder and that is has proper permissons.
#S3SYNCDIR = path to s3sync folder. This is where s3cmd.rb is located.
#I have not used the cache control. I don't use it cause caching slows my EC2 AMI.

BACKUPDIR=$bkupdir
SYNCDIR=/path/to/your/s3sync/folder/
S3BUCKET=$Bucket
S3PREFIX=$prefix
S3STORE=${S3BUCKET}:${S3PREFIX}

# move to the ruby sync direcotry where the .yml file is
# Also you can replace $RANDOM with anything you like.

cd ${SYNCDIR}
./s3sync.rb -r -v -p --cache-control="no-cache" ${BACKUPDIR} ${S3STORE} > $ldir/$mlog-$RANDOM.log

Semi Automated Downloads from S3 bucket. Data restoration

#!/bin/sh
# I have not tested this script with the EU buckets.
#This script will restore data from one bucket/prefix to your specifide folder.

# Please make sure that you credentials are correct. And remove ""
export AWS_ACCESS_KEY_ID="YOUR ACCESS KEY"
export AWS_SECRET_ACCESS_KEY="YOUR SECRET KEY"

#Enter all required info below. Make sure to double check everything that you specify.

echo -n "What directory would you like to restore your data to ? "
read bkupdir

echo -n "Please specify your bucket ?"
read Bucket

echo -n "Specify your prefix? "
read prefix

echo -n "Specify log name for this process. Please specify a time input "
read mlog

echo  -n "Specify log dir path. (plese specify a path to your log directory- Please include a / ): "
read ldir

#Please double check everything.
# Please specify the correct path to your s3sync folder.

BACKUPDIR=$bkupdir
SYNCDIR=/path/to/your/s3sync/folder/
S3BUCKET=$Bucket
S3PREFIX=$prefix
S3STORE=${S3BUCKET}:${S3PREFIX}

# move to the ruby sync direcotry where the .yml file is
cd ${SYNCDIR}
./s3sync.rb -r -v -p --cache-control="no-cache" ${S3STORE} ${BACKUPDIR} $ldir/$mlog-$RANDOM.log