Best Amazon S3 solution?
I'm evaluating the best/optimal solution to use Amazon S3:
Duplicity
s3fs
s3cmd
other
I'm testing Duplicity and it looks good!
Functional, secure, and fast … but I would like to browse the files too
What do you prefer?
13 Replies
We've used Jungle Disk some, and it may have all the features you seek (although it's commercial software).
When considering the "Best S3" solution, I'd make a distinction between backups and online "cheaper space." Obviously duplicity's forte is backups. Perhaps s3fs-FuseOverAmazon would be stronger for space due to its local cache (and lack of a built-in backup mechanism–but you could write to it with duplicity or perhaps rdiff-backup).
Edit: Wow, a new version of duplicity released yesterday--great to see the project becoming more active again.
s3fs =
At this time I was looking for opensource software.
I'm very pleasant with duplicity! Fast and Secure!
Yesterday I've tested s3fs and everything works smoothly and I can have the benefit of browsing files, but I lose security.
$ time dd if=/dev/zero of=/s3/test.20m bs=1024 count=20480
20480+0 records in
20480+0 records out
20971520 bytes (21 MB) copied, 76.7342 s, 273 kB/s
real 1m17.972s
user 0m0.015s
sys 0m0.172s
$ time cp /s3/test.20m ~
real 1m0.458s
user 0m0.000s
sys 0m0.045s
So, 273KB/s upload, 346KB/s download. Not a problem for a backup solution, but a pain in the *** when I'm actually trying to read one of my files. 346KB/s doesn't even max out a DSL modem! No, I did not use a DSL modem for this test. It was run on my linode in Fremont which apparently has lots of bandwidth available to it. And I'm using the latest version of s3fs.
If anyone's got a nice little trick to improve this situation, please let me know.
should save you bandwith & latency..
So your docroot can put to s3fs on each app server
> and not have to rsync and backup your data
Regardless of which provider(s) I were using to host my content, no matter how redundant their systems were, I'd still back everything up daily.
@hybinet:
At least for me, s3fs performance is pretty dismal…
If anyone's got a nice little trick to improve this situation, please let me know.
hybinet, did you ever manage to improve this ?
I'm looking at a similar setup with s3 (or fuse via rsync.net – ouch, pricey), but ~ 300 kB/s will sink it before it's even begun
Nope. I found s3fs simply too slow to be a replacement for SSHFS or NFS. Too much latency, too small bandwidth, and too much REST overhead.
It ain't exactly cheap, either, when you factor in all the costs of upload, download, storage, and per-transaction fees.
Drat! Back to the other options then.