IO tokens, performance, and distros

Hi,

The IO limiter is not bad, essentially you have an allocation of 'tokens' (my linode 256 has 800000 tokens at present), each time your linode makes an IO call, 1 token is used.

You can see how many tokens you have etc by cat'ing /proc/io_status, example:

njones@whio:~$ cat /proc/io_status
io_count=203884 io_rate=0 io_tokens=800000 token_refill=512 token_max=800000

~~[http://www.theshore.net/~caker/uml/patches/token-limiter.README" target="_blank">](http://www.theshore.net/~caker/uml/patc … ter.README">http://www.theshore.net/~caker/uml/patches/token-limiter.README]( has some decent explainations.

Not really sure about slackware sorry, I only got to the stage of installing it in a virtual machine at home and never using it. (You could see if there is anything on the Slackware website).

BZFlag: Not too sure. Could be okay

Perl/PHP/MySQL: All generally okay, the LAMP forum has some optimizations for stuff like MySQL:
* http://www.linode.com/forums/viewtopic.php?p=4570#4570

• http://www.linode.com/~caker/uml/my.cnf (Example MySQL Config)

• http://www.linode.com/wiki/index.php/SlashdotMyLinode (General tuning tips)

The token limiter is a pain sometimes – if you copy a big file, or grind the disk, it can choke you off, and when that happens, it's a really big inconvenience.

People complain about that, but they don't talk about the flipside. And the flipside is that no one else on your box ever sinks your linode. That's a big payoff. So while it is true that there's that big downside to the limiter, we do get something big in return.

Disks are a big problem in virtual environments, and it's not a problem that's specific to linode. What is specific to linode is their response, which tends to create winners and losers, I guess.

For me, it's a good response, because I almost never run out of io tokens (I haven't in more than a year), and the system keeps other people from choking me off.

But if you do a lot of disk intensive stuff, you might be someone who loses under this system.

I'd like to see a more complicated limiter, that takes into account the conditions on the host before choking you off. Although I know that writing such a thing would be a lot harder than writing the current limiter would be.

But the disks are the big bottleneck in a vps environment, and access to the disks is the most sparse resource in a vps setup. When I think through the way the token limiter works, it seems to me that it's unavoidably going to leave disk access capacity unused on the host.

Since that's the scarce resource that we run up against first, leaving unused capacity on the table seems unfortunate.

@astrashe2:

Disks are a big problem in virtual environments, and it's not a problem that's specific to linode.

regardless of price, what's the best technical solution to get rid of this problem? i.e., what set up would yield the best disk i/o for a bunch of linodes? feel free to replace any and all of caker's current infrastructure in your fantasy redesign.

I think it would be worth considering having many small hard disk, instead of a few big ones. My understanding is that each linode host has 3 physical hard disks in a raid-5 config (i could be remembering wrong though). There must be some cheap way to have a higher hard disk-to-linode ratio. That's the direction I'd go. Or faster non-mechanical drives, I think that's been discussed in other threads.

@besonen:

regardless of price, what's the best technical solution to get rid of this problem? i.e., what set up would yield the best disk i/o for a bunch of linodes? feel free to replace any and all of caker's current infrastructure in your fantasy redesign.

Fiber attached SAN service (eg EMC) using a SAN fabric and multiple GBIC interfaces. Migrating a linode between hosts would simply be a case of adjusting LUN allocation and detection on the relevant hosts; ie migration would take seconds. SAN storage servers are designed to multiplex traffic from many machines and have massively high throughput speeds.

It ain't cheap, though :-)

@kiomava:

I think it would be worth considering having many small hard disk, instead of a few big ones. My understanding is that each linode host has 3 physical hard disks in a raid-5 config (i could be remembering wrong though). There must be some cheap way to have a higher hard disk-to-linode ratio. That's the direction I'd go. Or faster non-mechanical drives, I think that's been discussed in other threads.
This page indicates that all hosts have two identical drives in a RAID 1 configuration. Personally, I prefer that it's mirrored rather than setup in a RAID 5, there is less chance of loss.

I do like the higher disk-to-linode ratio idea though, but no matter how high that is, it doesn't remove the need for the I/O Token Limiter patch. For what it's worth though, so long as I've had io_tokens, I've never had problems with disk performance, but I'd be willing to bet it could potentially speed up response times significantly.

I've often wondered if it would be possible to have readtokens and writetokens instead of just io_tokens. Writing to a disk is more expensive in terms of time and resources, but reading from a disk, expecially in a RAID1 configuration, is a lot less intensive.

Just a thought :-)

–deckert

@linvir:

Is a distro like Slackware a poor choice in terms of security? I'm assuming that I wouldn't be able to simply leave it alone like I do with my home machines. Is the constant manual updating a pain? Do you have to use things like slapt-get?

I use Slackware at home and on my Linode. When I started with Linode.com, the only option was Slackware v9, so that's what I'm currently running. I try to update things manually, which means I'm usually perusing the stable changelog. I'm definitely not a big supporter of Swaret, Slapt-get or any similar package managers, as they have the potential to break installations if your attention wavers, plus I'm an old-school Slacker and like the hands-on approach of Slackware.

Slackware is pretty solid in a default install. Keep in mind that when, during the install process, if you opt to have certain services installed, your choices impact the security posture of the install. Applications themselves are far less secure than kernelspace software (and the GNU toolset), so you need to be acutely aware of what you're installing (this goes for ALL distros), as Apache, for example, opens up a whole new world when dealing with security. I don't run any PHP-based software at all, because of the possible security implications. While Pat Volkerding has security in mind when compiling software packages, he's not the actual person developing the software…if the software code is loosely thrown together, it's certainly not his fault, although I'm pretty sure he evaluates each package that he includes in Slackware. He certainly updates each package if bugs or vulnerabilities are found by the software creator.

I've multiple layers of security applied to my Linode (layered security is the best approach). I've Snort running in daemon (IDS) mode (which reports all logged traffic to a machine on my home network), I've mod-security running and functioning as an application firewall for my web server. I've most services not running and those that do run are running on non-standard ports. I've IPTables implemented. I also have Denyhost blocking brute-force attempts. I've SSH configured to only accept key-based authentication. I've scripts running that logs certain stats to help me determine if my node has been compromised. All of this was done over the last 3 or so years.

While Slackware is a good candidate as a light yet secure distro, the admin who is installing and using plays a huge part of the overall security posture.

@Deckert:

I've often wondered if it would be possible to have readtokens and writetokens instead of just io_tokens. Writing to a disk is more expensive in terms of time and resources, but reading from a disk, expecially in a RAID1 configuration, is a lot less intensive.

I bet the expensive thing, when a bunch of people are trying to access their own little chunk of drive space, is seeking; the head moving from my "disk" to your "disk" to somebody else's. That's independent of whether we're reading or writing.

And I could be wrong, but I doubt RAID1 helps here either. It certainly can make reading twice as fast, as half the data can be read off each drive, but I don't see the system being smart enough to have the two drives doing completely different things. (And they'd have to meet up again for every write, anyway.)