Backing up Object Storage and downloading all files in OBJ bucket
- is there any need to take backup of object storage ?
- is there any way to download entire bucket ?
3 Replies
Our Object Storage infrastructure has a number of architectural protections in place to help keep your data safe. That said, these are resiliency measures rather than a proper backup mechanism, so I would nevertheless act as though you have a single copy of your Object Storage data.
As an S3-compatible service, Linode Object Storage supports bucket versioning so that you can retain previous copies of your bucket's objects. Even with this feature, backing up your Object Storage data outside of this system entirely is still a good call.
For backing up your Object Storage data externally, you may want to download its data to the SSD storage space of a Linode using our Backup Service. You may also want to consider using a custom backup solution, perhaps retaining the resulting backup data onto a Block Storage volume. Just so that you're aware, I'd like to note that our Backup Service only backs up the contents of your Linode's SSD, not its Block Storage data.
Like any backup system, each additional backup you make will cover for any failures that the other backups may experience: the more copies you have of your data, the more protection you give yourself against its loss.
With regards to downloading all of the objects in a bucket, I read through this Stack Overflow post, and it seems that using the
aws s3 sync
command will be the fastest way to do so. You may also consider usings3cmd sync
or Cyberduck to perform this operation.When using these utilities, you will need to adjust their endpoint URLs to reference our clusters and your bucket names. You will also need to create a key pair for these utilities to access your cluster. After this initial setup, they should have no troubles interfacing with our Object Storage systems using the S3 protocol.
Another option for backing up your Object Storage is to attach it to one of your Linodes using s3fs
with the help of this guide:
You may then create a rsync
backup of this data from your Linode like a regular directory:
Seriously?
Why is there not a simple checkbox next to the objects you want to download in order to do a multi-select download? That is waaaay less complicated to build than the suggestion above.
Multi-select download is a very common feature on virtually all websites.
You shouldn't need an AWS S3 dependency to do a mass download. That's ridiculous. I originally came to Linode as an alternative to AWS.
This seems like an intentional friction in order to make it as hard as possible for people to get their assets out of Linode Object Storage as possible, so they reconsider migrating away from this platform.