Object Storage - max folders?
I am writing a web application that uses S3/object storage to store encrypted files, user profile pictures etc…
I have a folder structure of something like bucket/<tenant_id>/<type_id>/<file.pdf.encrypted></file.pdf.encrypted></type_id></tenant_id>
Where file type could be something related to what the files are for and then files. The <type_id> could easily get in the 1000's+, is this going to be an issue?</type_id>
I am also using a shared bucket vs a bucket per tenant, any thoughts on that? I figure since all my files are encrypted and each tenant has their own encryption key it should be fine. :D
3 Replies
There's no limit on the number of folders, because folders technically don't exist. S3 and thus Linode Object Storage is a very simple key-value store. The key-value store doesn't care what the key is, so that you've structured it to look like a path is irrelevant.
There is a limit to the number of objects you can store in a cluster, which is 50 million. There's also a limit of 1,000 buckets per cluster. This guide indicates that these limits can be raised if needed by opening a support ticket.
I do something very similar, and went with the same approach - a single bucket, with a top-level “folder” per tenant.
I chose this route purely because of the bucket limit.
thanks, im going to stick with my single bucket and maybe reference which bucket is being used in my files table im using to track things on, then if my tenants combined hit that 50million limit, i can start using bucket 2 and life will go on.