SQL injection brings down VPS host
An ugly, ugly story, capped by the suicide of the owner/lead developer.
How glad I am that I'm with a provider whose developer is infinitely more clueful! Thanks Caker.
22 Replies
There was a local "host" who was mostly just a crew of web designers who were renting 2 servers in a colo. They never backed anything up, even from their local workstations. They thought a set of mirrored drives in the server was sufficient. The file system on one of the servers became corrupt and they were hosed. They decided to just shut their doors as they realized, after screwing over 3/4 of their customers, they didn't have the know how to manage servers or web sites, just create them. Granted they were able to recover a lot of their data because it was laying around on workstations and such.
> Some 50 percent of Vaserv's customers signed up for unmanaged service, which doesn't include data backup, Foster said. It remains unclear of those website owners will ever be able to retrieve their lost data, he said.
This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.
@waldo:
This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.
Well put. Every story like this is a message to the whole world: data loss is coming for you. Be backed up.
I've been with VAServ a couple of times for little projects. Their service was pretty good, especially for the price. (Compared to other "ordinary" hosts, of course. Linode isn't an "ordinary" host!) Too bad they suffered this massive outage due to a piece of software they had virtually no control over.
VAServ seems to be recovering pretty well, at least for now. I hope they survive this mess. Since they have a pretty loyal customer base, hopefully it wouldn't be too difficult. On the other hand, the Indian guy who developed/sold HyperVM (the control panel software which is deemed responsible for the vulnerability) committed suicide, and a lot of other "ordinary" hosts have been using it too… so there's definitely a storm coming over there.
Lessons
1. Use free and open-source software whenever you can. Obfuscated PHP to prevent copying? Gimme a break.
2. If you create an in-house solution instead, do it well (like Linode!)
@waldo:
This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.
Yup.
And never rely only on your webhost's backup service, not even at Linode. What good is a backup if it's in the same datacenter? Backup, backup again, backup your backups, and backup the backup of your backups!
@hybinet:
And never rely only on your webhost's backup service, not even at Linode. What good is a backup if it's in the same datacenter? Backup, backup again, backup your backups, and backup the backup of your backups!
And another step a lot of people forget, test your backups. What good is a backup if it's corrupt or contains corrupt data?
I knew someone who was "backuping" their database on a regular basis, even taking the backups off site. Everything looked great. Until one day they needed to recover, turns out because of some file-locking issue the database hadn't actually been backed up….
They had only tested when the system was first put into place, everything worked and looked great. At some point over the years something stopped working, but they didn't even do yearly tests, let alone quarterly, monthly or weekly.
Fortunately for them the data they needed was for research purposes and not to recover because of failure.
@hybinet:
Lessons
1. Use free and open-source software whenever you can. Obfuscated PHP to prevent copying? Gimme a break.
2. If you create an in-house solution instead, do it well (like Linode!)
3. Don't use a provider that knowingly aided and abetted phishers and other types.
I used to host at linode and switched to fsckvps in January? because it saved me $10/month.
…
Hey look, I'm back, not much richer and tons more frustrated.
Luckily for me, my data was saved.
Ughhhhh
@tim101:
I used to host at linode and switched to fsckvps in January? because it saved me $10/month.
You Get What You Pay For ™
Honestly though, fsckvps was pretty good for $9.95/mo, at least when compared to other sub-$10 providers (of which there are more than a handful).
The question is: Do you want $9.95 worth of service, or do you want $19.95 worth of service? There's a good reason why Linode refuses to offer anything below that price point, even though there may very well be a market for something like Linode 180.
@hybinet:
Honestly though, fsckvps was pretty good for $9.95/mo, at least when compared to other sub-$10 providers (of which there are more than a handful). Quite correct. Really, to be honest, fsck was great until the HyperVM attack.
:(
HyperVM itself was inferior to Linode's control panel- but really, I didn't have to mess with it very much.
> The question is: Do you want $9.95 worth of service, or do you want $19.95 worth of service?
I have learned my lesson, as I bet many others have too.
It took a good 6-7 hours work after 48 hours downtime to get everything online over here. I had backups but they were not recent (another lesson learned!-I guess if the worst happened I would have had something though). In the end I had to wait for them to get http online and hack (irony time) a script that I made to run the command through php to create a tar of my files and transfer here. Ssh (so sftp) is still offline, so I couldn't download files the normal way. This was on my server anyway, they are not really responding to tickets.
I feel for less technical fsck-ers & those who lost data.
Anyway, I'm glad to be live & back at linode
@waldo:
And another step a lot of people forget, test your backups. What good is a backup if it's corrupt or contains corrupt data?
I knew someone who was "backuping" their database on a regular basis, even taking the backups off site. Everything looked great. Until one day they needed to recover, turns out because of some file-locking issue the database hadn't actually been backed up….
They had only tested when the system was first put into place, everything worked and looked great. At some point over the years something stopped working, but they didn't even do yearly tests, let alone quarterly, monthly or weekly.
Fortunately for them the data they needed was for research purposes and not to recover because of failure.
Yes, backing up is simple, it's the restoring which tends to get hard… Remember an old joke about announcement of revolutionary compression software able to compress any data to 100 bytes. Now developers started work on the decompression part…
@tim101:
It took a good 6-7 hours work after 48 hours downtime to get everything online over here.
When you add up how much that is worth to you in lost time/revenue etc the $10 a month saving would be blown away for a few years
Good to see you back!
@neo:
Remember an old joke about announcement of revolutionary compression software able to compress any data to 100 bytes. Now developers started work on the decompression part…
LOL
Well, it ain't just a joke. MD5 can "compress" anything to 16 bytes, and it's considered "weak" so theoretically it should be possible to "decompress" the hash to its original representation… Just bruteforce it for a few weeks and look for a collision!
I'm glad all my other stuff is on Linode. I really ought to get that backup script working again though…
Edit: Just remembered, I had considered applying for a job vacancy they had at a2b2/cheapvps previously (part of vaserv.) Probably a good thing I didn't!
I am giving Linode a try because my current VPS host has only HyperVM for things like changing or reinstalling the Linux distro. I was liking the combo of HyperVM and LxAdmin until the VAServ hack. My host then took HyperVM and LxAdmin offline. Since those products will never be patched now, that host is already out of business, they just don't know it yet.
So, in this thread, and a couple of reviews on the web, people have asserted that the in-house Linode Manager is great, secure, well-made, etc.
Can anyone cite proof to help me feel better that an in-house product of a 6-year old company, who also has a lot of non-software development responsibilities, can create a secure, solid piece of software? Have independent groups or labs tried to break it?
I am shell-shocked by this whole VAServ incident and don't want to jump from the frying pan into a different frying pan.
Thanks!
That's got to be worth something.
@Guspaz:
… They'll never have the problem of their vendor (HyperVM) up and disappearing, since Linode is also responsible for the development of it. …
Totally agree with Guspaz, in-house software development—at least in this case—is really a plus.
To be honest with you, we are not yet sure whether Linode is more/less secure than HyperVM. HyperVM was a great product, and I wish that it goes open-source and then with the help of the community it will be the ultimate VPS manager for OpenVZ and XEN.
Linode is a great product and I am enjoying my stay here, yet I wish if I can use it with other ISPs.
@mat989:
I am sure that a lot are willing to license the software…
A lot of people say great things about the LPM, including its ease of use, and it's a great advantage that Linode has over other companies.
That being the case, it's not to Linode's advantage to license it, so I don't expect that'll change.
(Disclaimer: Not an employee of Linode, not speaking for anyone at all, just my own views as a customer)