@taupehat:

My node is pared down to the basics: apache, php, mysql, postfix, imap(cyrus), and amavis

Assuming you have a Linode 64 (usually where the swap-thrashers are), that's a mouth-full. Putting a LAMP setup on a Linode 64 has always been pushing the limits… putting that aside, you also put on imap (tends to spawn some memory crunchers with lots of clients connecting), and the big one: amavis. Any mail filtering/scanning software eats away resources, no Linode 64 can handle a LAMP install alongside either spam filtering or virus scanning from what I've seen. I wouldn't even run your configuration on a Linode 96.

It's not hard to tell how your setup started thrashing. My apologies if you are in fact not on a 64 or 96, it had never been mentioned how much ram your configuration has, which is a rather important detail with your configuration.

You could change from Apache to something like lighttpd

You could move MySQL off of your site and subscribe to the (currently beta) MySQL VAS (Value Added Service) that is free right now http://www.linode.com/forums/viewtopic.php?t=1460

Those are two of the easiest things that you can DoRightNow™ to attempt to lower your overall memory usage. Though like tierra mentioned trying to do your own spam/av filtering is rough stuff especially on a L64.

adamg (the guy behind the MySQL VAS) has been working up a similar service for spam/av and quarantine but it's not ready to be in beta yet. But at least you know that these types of services are in the pipe getting ready to come down to you.

When your system starts to swap like crazy, you should log on and run top, and find out which processes are using the most cpu time, swap space, memory, etc…

once you can pinpoint the culprit, it will be a lot easier to make it behave.

It's been a while since I ran Cyrus, but you might find Courier-IMAP to be more lightweight.

A while ago I outsourced the spam and antivirus stuff to Tucows. It's been working fine (and it's cheap), and it would eliminate the biggest resource hog from your system.

Good luck!

Paul

Looks like I got it down to about 15 megs of swap on average. Took out amavis, and am relying on a good healthy set of RBLs to keep the spam down - and that's actually working pretty well.

Right now, mysql is by far and away the heaviest user of RAM on my system, starting at 14 megs, and gradually increasing. I try to restart it at least once a day, and have put in a request to see about that external server.

That said, this has proven to be an interesting experience. I'm beginning to come to the conclusion that 64 megs of ram isn't enough to do anything useful on a webserver. To me, a minimal server is LAMP + email. I can't imagine what a point of presense on the internet would be useful for with less.

I've got my 64 running quite well…

Mem: 59356k total, 48600k used, 10756k free, 7528k buffers

Swap: 263160k total, 5668k used, 257492k free, 21288k cached

Postfix, apache with php, pop3, mysql, named… etc…

I dunno, trim your configs.

-tiz

Amavis was my main pain. I moved it off to another host and had postfix relay through it and everything got better.

My biggest problem is spamd and heavy spam attacks. If there's heavy spam volume it also tends to use mysql (whitelist/bayes data in my config is in it) and takes the load up, iotokens down.

My solution has been to use the watchtokens.pl script I have created (see http://www.kotalampi.com/scripts/watchtokens ). It checks your available tokens and system load and if they reach critical levels (read the script), it runs the script /etc/rc.d/stop.sh and when things are back to normal, it runs /etc/rc.d/start.sh. It also creates stats to /var/adm/watchtokens.$DATE file (that rotates daily) and you can tail -f it to see what is going on.

In my stop.sh I block port 25 and all new connection requests. In start.sh I allow requests again. I have secondary MX so this has worked really well for me. This has proven out to be a good way to throttle the load and avoid iotokens to run totally out.

My stop.sh:

!/bin/sh

/sbin/iptables -D INPUT -i eth0 -p tcp -m tcp –dport 25 -j ACCEPT

My start.sh:

!/bin/sh

/sbin/iptables -A INPUT -i eth0 -p tcp -m tcp --dport 25 -j ACCEPT

The script can be started like this:

/usr/local/bin/watchtokens.pl --sleep=15 --stop=yes &

Sleep option tells you how often you want to check iotokens, stop-option tells if you really want to run stop.sh/start.sh or just log the data.

Cheers,

Risto

That URL seems to be broken.

Yes, because it looks like phpBB added ').' to the end of the URL, let's try again: http://www.kotalampi.com/scripts/watchtokens

RE your mysql memory usage, have you tried fiddling with /etc/my.cnf? If you are using innodb tables, the setting for innodbbufferpool_size can be quite important to overall RAM usage.

I'm not using innodb… it's such a disk hog. I have tuned up other variables in my.cnf though. Things like :

query-cache-type=1

query-cache-size=16M

key_buffer=128k

sortbuffersize=256K

netbufferlength=16K

thread_stack=256K

These might not be optimal but mysql isn't as big thrasher as spamd.

Risto