Creative ways to extract more disk space from a Linode?

I think your issue with s3 is that you're using it through s3fs. You should really find some gallery software that interacts directly with s3. I can't really make any suggestions because I couldn't find one when I last tried to look.

I'd suggest you supplement your linode with a smugmug account. The cheapest plan is $5 a month and you can store unlimited (*) number of jpegs. They're backed by s3 if that means anything. If you don't like smugmug there's also phanfare.

(*) pretty much unlimited

@sirpengi:

I think your issue with s3 is that you're using it through s3fs. You should really find some gallery software that interacts directly with s3. I can't really make any suggestions because I couldn't find one when I last tried to look.

I'd suggest you supplement your linode with a smugmug account. The cheapest plan is $5 a month and you can store unlimited (*) number of jpegs. They're backed by s3 if that means anything. If you don't like smugmug there's also phanfare.

(*) pretty much unlimited

Thanks, those are good suggestions. I did try to use S3 in a better integrated manner with gallery but I found that the only instructions I found for doing so were a) really complicated (but doable), and b) required the use of mysql as the database backend. I already use postgresql for my gallery config so without major headache of switching databases, or hours upon hours of altering gallery to work with S3 and postgresql, I can't easily use that solution.

Perhaps despite the hours and hours of effort that will be required on my part, this is the best solution. But I would still be interested in learning about other possible solutions.

A compressed filesystem wouldn't help you much, not if most of your usage (especially the part that keeps growing) is photos. Assuming those are JPGs, further compression is not likely to be possible, and you'll just have a lot of complexity for little or no (or possibly negative!) return.

I finally gave up and switched to a dedicated Atom server for $50/month from Wholesale Internet. It has a 500GB hard drive. True, the CPU isn't the same as what I had with Linode, but that's okay with me. It may not be for you, but I do find that it handles everything I threw at my Linode much better than the Linode did, because the disk performance is so much better. And I never even have to think about running out of disk space.

@bji:

Has anyone played with compressed filesystems under Linux? I did a little searching but the results were not encouraging; it doesn't look like there are many 'seamless' options for doing this, and using a compressed filesystem has the added worry of possibly interacting badly with Linode's virtual disk management tools.
Given that most of your space is going to photos, a compressed filesystem is unlikely to help much, since your data is already compressed.

> Any other creative ideas? Anything with the bandwidth and latency problems similar to S3 is basically out of the question, so most if not all network based storage solutions are probably not feasable - unless the networked storage was local to the data center.
Is there any opportunity for you to shard your albums by frequency of access to separate frequently accessed stuff from less so? If I were in your shoes, I might consider storing the less frequently accessed stuff right at home (essentially free, unlimited storage), with a network mounted filesystem (preferably over a secure VPN tunnel from the home server to my Linode), and then push older or less accessed stuff down there.

My guess is that growth is slow enough that you could manage the division of photos manually, say selecting the top 'n' oldest albums once a month and replacing them on your Linode with symbolic links to the network mount from home. Were I doing this I'd probably have an automated sync script always keeping the home storage completely in sync with anything stored locally on the Linode so when you need to shift something, it's just removing the local copy and making the link to the matching directory already present over the network share.

I'm sure there must be some other more automated way (say a generic caching layer above the network filesystem), but I'd bet some simple manual management would be fine, and you'd still have the basic Linux filesystem cache for short term accesses.

Yes, this would mean that accesses to the older stuff would make your Linode pull the data from your house, and then push it to the client but much of that I/O should overlap so the timing need not be much worse than the basic retrieval from home, and it's for less frequently accessed stuff. And given that the images would have existed initially on the Linode before being shuffled off home, gallery's thumbnails/cache could still exist locally. And yes, if your home's link or server went down you'd lose access to those photos for the outage.

Now this may not be practical for any number of reasons (everything accessed frequently, unbearably slow or unstable link at home, etc…) but it may be worth considering or at least testing.

-- David

s3fs sucks hard, as was made pretty clear in the last thread you opened. That's not necessarily a fault in s3fs, since S3 itself was never designed for filesystem-like access patterns.

Have you tried buying a second VPS in the same geographical region, but from a cheap provider that has lots of disk space? Although NFS or SSHFS would also involve network latency and bandwidth limits, they usually perform much better than S3 because those protocols were designed to be filesystems in the first place.

Now, here's my $0.02 just in case you're considering any further modifications to your existing gallery app: Don't. Those hours would be much better spent looking for a new gallery app that either comes with decent S3 integration by default, or has a well-tested plug-in to the same effect. Besides, Google can find you ways to migrate data from most open-source apps to most other open-source apps, though it's not always easy. At any rate, you'd be investing your time in a semi-permanent solution rather than in just another stopgap measure. If you don't have a very compelling reason to stick to PostgreSQL, look into alternatives that use MySQL – it's not that bad. Besides, you can run both on the same server if you need to.

Or just get coffee and/or lunch from a slightly cheaper place a couple of times a month, and use the extra cash to buy a bigger Linode 8)

Using a compressed filesystem on images is not going to work. Give up on that solution.

You seem determined to use the Linode when it possibly isn't the best option. However if you're too paranoid to let some dedicated photo hosting service host it for you then upgrade that Linode.

@Xan:

It may not be for you, but I do find that it handles everything I threw at my Linode much better than the Linode did, because the disk performance is so much better.

I find it very hard to believe that your $50 Atom dedicated server has better disk performance than a Linode running on 15K RPM SAS drives in RAID10. Unless there was a heavy user on your linode, in which case a simple ticket would have solved that. For that matter, a Linode should trivially outperform an Atom in the CPU department too.

My $20 linode certainly outperforms any low-end dedicated server I've ever had, even when the dedicated server cost four times as much.

It's cheating like hell but has anyone tried gmailfs on a linode?

It's free, should provide plenty of space, and should be no slower than s3fs. It's an ugly solution but might just work.

The only other solution I can think of is put thumbnails and application code on the linode and bigger images on some form of cheap web hosting.

@Guspaz:

@Xan:

It may not be for you, but I do find that it handles everything I threw at my Linode much better than the Linode did, because the disk performance is so much better.

I find it very hard to believe that your $50 Atom dedicated server has better disk performance than a Linode running on 15K RPM SAS drives in RAID10. Unless there was a heavy user on your linode, in which case a simple ticket would have solved that. For that matter, a Linode should trivially outperform an Atom in the CPU department too.

My $20 linode certainly outperforms any low-end dedicated server I've ever had, even when the dedicated server cost four times as much.

Where it was most noticeable was email. I keep large email folders in maildir format on the server. Doing anything with email, whether it's just listing a folder, moving messages, or whatever, is much faster on the dedicated box where I have my own hard drive. I don't think it was a problem with a rogue user; I was on Linode for a very long time, and did move from host to host on occasion.

Surely Linode is a better deal on CPU, but that wasn't my bottleneck. The Atom is more than enough for what I need to do. And that does include a photo gallery with server-side resizing.

I've had the opposite experience. Disk performance, both in terms of sequential transfer and IOPS, is far higher on my Linode than my dedicated server's 80GB 7200RPM drive was.

@-Alex-:

Using a compressed filesystem on images is not going to work. Give up on that solution.

You seem determined to use the Linode when it possibly isn't the best option. However if you're too paranoid to let some dedicated photo hosting service host it for you then upgrade that Linode.

I am using Linode for this because we have 8+ years of photo albums stored in gallery and it would be a monumental task to convert all of that over to some other service. It has nothing to do with paranoia.

It's getting to the point where I have no more outs though and just have to pony up more $$; because the only other options all involve hours upon hours of effort on my part.

What I was really hoping for was some networked storage service that is local in the Linode Dallas data center that I could host networked storage on for cheap-ish and still get good performance from my Linode. Barring that, it's going to have to be a Linode upgrade …

Of course I could always hope for Linode to upgrade their disk space plans … :cry:

Well since you seem to be open to suggestions. How reduced in size and compression/quality are the full size images?

Is this Gallery 2 you're using? If so, do you have more than 1 resized version? If you're open to changing what you host and how things are organized, you can eliminate the resized version and reduce the quality of the full size version so you just have a thumb and a "full-size" image that loads quickly. Of course if you're using this as your archive to that's not going to work, but I'd suggest since you're bumping up against disk space issues and if you're trying to use Linode/Gallery as your photo archive, that you reconsider that. It's cheaper to use another online backup service such as Mozy or Crashplan for backups.

@Guspaz:

I've had the opposite experience. Disk performance, both in terms of sequential transfer and IOPS, is far higher on my Linode than my dedicated server's 80GB 7200RPM drive was.

This is exactly my experience too. I had a 5600+ dedicated server from hetzner and disk IO was always way faster at linode.

I also had a cheap dedicated server from bytemark and the linode won on everything but disk space.

If you use a program like Drupal, you can have it integrate with S3 without mounting a S3 bucket as a partition and thus move all of your photos your Linode.

See http://drupal.org/project/storageapi and http://drupal.org/project/mediamover to start

@bji:

Hello all. You may remember me as the grumpy guy

I remember saying that Linode's the wrong place for storing pictures. ;-) However, since you're asking for alternatives, why not find a few likeminded individuals, rent one of them $50 Atom servers with a 500GB disk and resell that space with simple NFS exports (or whatever remote FS blows your hair back).

If you sell each 100GB for $10/m, four of your clients can each get a very good deal on 100GB of storage and you'll be out of pocket $10/m for the last 100GB of storage (minus small overheads, of course). Bonus points if you get the server in the same DC where Linode hosts.

Me, I "host" all my pictures with Google's Picasaweb. For 20GB of storage I pay $5 a year and get all the other goodness along with it (location tagging, free bandwidth, easy sharing and exceptionally easy uploading and management of that process).

–deckert

But unless you hack the gallery code to emit hotlinks to the storageserver, you're paying for twice the bandwidth. That Atom machine won't be on Linode's internal free-transfer network.

That assumes that a personal photo gallery would be bandwidth limited.

I still think the best option would be for Linode to offer a SAN service accessible over the private network. Customers get some very high performance local storage, and then can affordably supplement that with cheap lower performance network bulk storage.

If you mostly need photo storage, consider google's Picasa + paying for extra space.

It works out to ~2c/gb/mo - much cheaper than rackspace or S3. Of course it doesn't have the same flexibility, though..

https://www.google.com/accounts/PurchaseStorage

80 GB = ($20.00 USD per year)

@AviMarcus:

If you mostly need photo storage, consider google's Picasa + paying for extra space.

It works out to ~2c/gb/mo - much cheaper than rackspace or S3. Of course it doesn't have the same flexibility, though..

https://www.google.com/accounts/PurchaseStorage

80 GB = ($20.00 USD per year)

Is there a way to transfer Gallery photo albums into Picasa, retaining all of the structure and markup that was present in Gallery?

I've been using Picasa to manage my photos. It's OK, but I have two main complaints. First, it seems extremely difficult to manage your photos through the web interface, only the desktop app is practical for that. I couldn't even figure out how to move multiple photos between albums. Maybe there's a way, but it certainly isn't by dragging, they don't make it easy.

Second, I'm always hesitant to show people photos on my Picasa album, because it shows people super low bitrate JPEGs of your photos. Parts of my photos that, when viewing the original image, have a pleasant grain, instead get blocky artifacting when people view it on Picasa.

@bji:

Is there a way to transfer Gallery photo albums into Picasa, retaining all of the structure and markup that was present in Gallery?
I doubt it…

@Guspaz:

Second, I'm always hesitant to show people photos on my Picasa album, because it shows people super low bitrate JPEGs of your photos. Parts of my photos that, when viewing the original image, have a pleasant grain, instead get blocky artifacting when people view it on Picasa.

Picasa asks you what size you want to upload. I've had no problems with that. Perhaps you should try uploading at a higher resolution?

Picasa does have a "sync" feature, but otherwise, I've not tried to move stuff around too much.

I uploaded the full resolution version, and I can see it through Picasa. The problem is the picasa web viewer, which does dynamic resizing, outputs extremely low quality JPEGs. So the user sees a very poor quality version unless they zoom all the way into the image. As such, Picasa is pretty bad for showing your photos to people.

Well thanks for all of the great suggestions. For the moment the best solution for me was to upgrade to a Linode 1024. While I would love to move my very costly gallery2 site to some other service, nothing duplicates gallery2's functionality and moreover, it would be very difficult to migrate our existing albums to another service. Believe me, if I knew that gallery2 would end up holding me hostage in this way I never would have chosen it for hosting my photos in the first place …

I would seriously consider any fast-ish local network storage available to a Linode in the Dallas datacenter, but haven't heard of any yet.

And I would still really appreciate it if Linode could upgrade their disk plans. Disks have twice the capacity for the cost that they had a year or two ago but Linodes still have the same space for the same cost. And please do not take this as an opportunity to try to educate me on why Linode disks cost more than consumer hard drives; I know about all of the great reliability features of Linode disks, but I also know that with all 'fixed' costs aside, cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Oh - and in praise of Linode, this is twice in the past two weeks that I've upgraded my Linode (from a 512 to a 768 to a 1024) and both times it was so seamless and painless.

@bji:

cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Do you buy a new hard drive every time prices fall?

@bji:

And I would still really appreciate it if Linode could upgrade their disk plans. Disks have twice the capacity for the cost that they had a year or two ago but Linodes still have the same space for the same cost. And please do not take this as an opportunity to try to educate me on why Linode disks cost more than consumer hard drives; I know about all of the great reliability features of Linode disks, but I also know that with all 'fixed' costs aside, cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Then perhaps we can educate you on the costs involved in upgrading every single legacy host… While newer drives are certainly cheaper, those drives don't magically appear in the thousands of hosts Linode maintains.

@JshWright:

@bji:

And I would still really appreciate it if Linode could upgrade their disk plans. Disks have twice the capacity for the cost that they had a year or two ago but Linodes still have the same space for the same cost. And please do not take this as an opportunity to try to educate me on why Linode disks cost more than consumer hard drives; I know about all of the great reliability features of Linode disks, but I also know that with all 'fixed' costs aside, cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Then perhaps we can educate you on the costs involved in upgrading every single legacy host… While newer drives are certainly cheaper, those drives don't magically appear in the thousands of hosts Linode maintains.

Then why not have newer hosts come with more storage and allow people to pay a reasonable amount to upgrade to the higher storage plan? Meaning why not provide an 'upgrade' to a Linode 768 that provides 1.5X or even 2X the storage but requires migrating to a new host?

Thinking that all hosts have to be upgraded at the same time is not thinking very creatively about the problem.

@glg:

@bji:

cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Do you buy a new hard drive every time prices fall?

If I answer 'yes', how will that be relevent to the fundamental point of my post?

If I answer 'no', how will that be relevent to the fundamental point of my post?

FYI, in the past, Linode has offered disk space upgrades when not all older hosts have the capacity for it, offering migrations to other hosts when necessary.

I am confident that Linode is aware of how much everyone wants more disk space and will offer it when feasible.

Edit: Thanks to the magic of RAID, they could upgrade a host to larger drives live.

@bji:

@glg:

@bji:

cost per unit disk space for drives falls faster than Linode's prices for disk space ever have.

Do you buy a new hard drive every time prices fall?

If I answer 'yes', how will that be relevent to the fundamental point of my post?

If I answer 'no', how will that be relevent to the fundamental point of my post?

I know that the real answer is no, so why would you expect linode to magically have the money to buy new disks every couple months?

@bji:

Then why not have newer hosts come with more storage and allow people to pay a reasonable amount to upgrade to the higher storage plan? Meaning why not provide an 'upgrade' to a Linode 768 that provides 1.5X or even 2X the storage but requires migrating to a new host?

This already exists, upgrade to the next plan. Further fragmenting their plans would be a disaster management-wise and marketing-wise.

The aggregate computing power (i.e. the number of cores) and amount of RAM that can be fit in one server are also increasing. So, odds are good that the monumental increases in disk capacity – 600 GB drives are now commercially available! -- are being offset by an increase in the number of Linodes per server.

Also, the rack space previously reserved for large network-accessible storage arrays may have to be used to install mammoth routers capable of handling individual routes for each Linode in a datacenter; this is due to the revolt in the IPv6 thread.

And some day they'll have to move to SSDs just to keep up, because as CPU and RAM resources increase by Moore's law, IOPS will increasingly become the bottleneck. They won't be able to fit more linodes onto a host regardless of CPU or RAM availability because the IO won't be there to support them. And that certainly won't help capacity whenever that happens.

Cheap and slow supplemental network storage may be the only real answer within the next few years. And I don't think it requires quite as much rack space as hoopycat implies, when you can fit 36TB in 2U, 144TB in 4U, and with multiple vendors having just announced 1TB per platter, and 5 platters being possible in 3.5" drives, that's going to go up a bunch.

There are many other concerns, certainly. Manpower, performance, reliability, cost, etc. I don't think rack space is a primary concern, though.

@glg:

@bji:

@glg:

Do you buy a new hard drive every time prices fall?

If I answer 'yes', how will that be relevent to the fundamental point of my post?

If I answer 'no', how will that be relevent to the fundamental point of my post?

I know that the real answer is no, so why would you expect linode to magically have the money to buy new disks every couple months?

Straw man. I never said that Linode had to upgrade their disks every couple of months, nor did I say that they had to upgrade all of their disks at the same time.

As I have harped on many times in the past, Linode's disk offerings were small to begin with (yes I bought a Linode when the offering was 1 GB per Linode disk space) and even then haven't kept pace, in my opinion, with the cost of drives.

That being said, I think there are business reasons why Linode doesn't offer more disk space; those being that the majority of the customers come to Linode because of the high CPU allotments, good control panel, and support. Disk space only factors in for a minority such as myself.

That doesn't meant that I can't still want it, can't still believe that if Linode wanted to prioritize disk space that it couldn't do so, and can't still make my opinions known in the hope that Linode takes action.

@glg:

@bji:

Then why not have newer hosts come with more storage and allow people to pay a reasonable amount to upgrade to the higher storage plan? Meaning why not provide an 'upgrade' to a Linode 768 that provides 1.5X or even 2X the storage but requires migrating to a new host?

This already exists, upgrade to the next plan. Further fragmenting their plans would be a disaster management-wise and marketing-wise.

That is not a very compelling upgrade path. I know, because I recently had to do it; I'm paying an extra $10 per month to get a bunch of extra CPU and RAM that I don't need, just so that I can get 8 GB more disk. $120 per year for 8 GB of disk seems too expensive to me.

Right now I'm a customer who desperately wants to find an alternative to Linode for my hosting needs. I would rather be a customer who is happy to stay with Linode. Linode may not care whether or not some customers are desperate to leave or happy to stay, but I think they should be.

@bji:

As I have harped on many times in the past, Linode's disk offerings were small to begin with (yes I bought a Linode when the offering was 1 GB per Linode disk space) and even then haven't kept pace, in my opinion, with the cost of drives.

Disk space has become dirt cheap in the last few years but disk performance is still very expensive. Shared hosting isn't going to work with anything but the fastest disks and everything needs to be RAID-1 at a minimum. I'd love more disk space too but if Linode went with EMC or HP SAN's they would be able to give us maybe twice the disk space for 5 times the cost.

If you need lots of disk space a bottom end dedicated server with a real disk might be better for you. I don't know if having one in Germany is any good for you but I've used netdirekt.de in the past and they are good value for money. Linode wins at everything except disk capacity.

@sednet:

Linode wins at everything except disk capacity.

I could not agree with you more. Some people may think I am greedy, but I want Linode to win at everything!

I've seen threads going back to 2009 where people are begging for internal network storage of any kind (fast or slow). I don't understand the hold up. I have a website that I've run for years as a hobby, I'm out of drive space so my only option is shut down or rewrite code.

I am otherwise so impressed with Linode, that I'd rather start a new website than switch hosts.

@bji:

@sednet:

Linode wins at everything except disk capacity.

I could not agree with you more. Some people may think I am greedy, but I want Linode to win at everything!

I always thought that they kept it small to be able to quickly transfer linodes between data centers. If they offered everyone 300GB disks, datacenter -> datacenter or 512->768 would not be reasonable.

Which is why SANs would be nice, since migrating around inside the datacenter (linode moves, resizes, etc) wouldn't need to actually move, since it's external storage.

One potential option for extracting more space out of your linode: filesystem-level compression. Depending on the type of content you're storing, it could be useful, and Linodes tend to have an abundance of processing power.

Nearly all, VPS players have had or do have problems/fluctuations with their SANS setup including Rackspace, UK2, Softlayer, Amazon quassi-San style storage, etc… Linode doesnt extreme HD issues with regards to performance, lets leave it that way

It's easy to not have storage issues when you don't have any storage. :-\

@reknirtved:

Nearly all, VPS players have had or do have problems/fluctuations with their SANS setup including Rackspace, UK2, Softlayer, Amazon quassi-San style storage, etc… Linode doesnt extreme HD issues with regards to performance, lets leave it that way

This is part of the reason I have requested SAN as an addon. Let the OS run off the local drives that have higher performance and let me mount an additional SAN drive for more storage. That would also help the SAN storage since you wouldn't have every Linode reading every file off the SAN.