Store huge number of small files
I need to store, read and write huge number of small files, about a million files of 1kB, one file at a time.
What will be the best way to store them, in xfs, ext4 file system or maybe in tar or perhaps you have other suggestion?
5 Replies
eruzanski
Linode Staff
Linode Object Storage is the ideal solution perfectly suited for your specific use case. To efficiently store, read, and write a vast quantity of small files, including approximately one million files each at 1kB in size, Object Storage is the ultimate choice.
I highly recommend delving into the product documentation for Object Storage to ensure it fully aligns with your unique requirements:
I think it'll depend on your use case as to whether or not Object Storage is the best option. I'm using it for delivering map tiles (thousands of small image files) and it's been reasonably reliable, although the Frankfurt data centre has had a worrying number of outages and on-going access issues. If you're regularly creating/reading/updating your files, the lag introduced by non-local storage could be a problem for you and you'll also need to factor in the possibility that you'll experience outages (check the historic Linode status reports to make sure the data centre you're thinking of using is reliable).
I understand how to create block storage and/or object storage but I don't know what URL you would give someone to access a file (like a PDF) you have on one or the other? Can someone explain?
I think using various databases might be the best option for storing small files. Regarding another question about accessing PDF files, I can say that for working with such files, I use the online editor https://pdfguru.com/azw3-to-pdf which allows you to convert AZW3 to PDF, among many other formats. It's also indispensable for editing, merging, and splitting PDF files. Moreover, it works on almost all operating systems. You might find it useful.