A (hopefully useful) Summary of Memory Tweaks

Good advice. I just want to chime in with a little non-spamassassin anti-spam advice.

Require that a connecting machine introduce itself with HELO. Require that it give you a FQDN (as in the RFC). Block if it tells you it's you; a lot of spam introduces itself as YOUR hostname or IP address.

Require that it wait for a greeting before beginning to send its message. Introduce a delay in order to trap poorly-written SMTP bots.

You'd be amazed how much spam doesn't follow the simple protocol rules.

Another thing you can do is SMTP callouts. This is easy on exim, which is Debian's default. I'm not sure about others. The idea here is that after you've heard the From: line, you make a connection to the sender's mail server and verify that the sender actually exists. If not, the incoming connection is dropped.

All these allow you to drop spam early in the cycle, before they've even sent you the body of the message. It's easy on the old CPU/RAM!

mikegrb mentioned graylisting in another thread. It's a really lightweight spam fighting technique; I've implemented it on my linode 80 and the performance hit, if any, is unnoticeable.

I've seen some other discussions now on spam-fighting techniques with limited resources. But, along the lines of the discussion in the original post, comparing alternative choices for applications, is there particular mail software (or array of software) that is uniquely good on limited resources?

I'm on a Linode80 and currently using Qmail along with Courier for IMAP. Now, aside from the barrage of spam, I don't have a lot of email users, so the traffic isn't heavy. IMAP is only used for webmail. I'm running Debian, if that matters.

I'm not sure about greylisting, myself. It sounds like a great idea and I'm sure it works quite well, at least most of the time.

But it makes assumptions about the way email works that you really can't make. Why can't a re-send come from a different IP? Why can't there be a unique sender, if I'm running a list and want it that way?

So there's a number of sites you can't get mail from, and you can't really complain because they're not violating any spec. So you end up having to maintain a whitelist.

I'd definitely try the HELO / callout methods before I'd resort to greylisting (or to relying on blocklists, but that's another story).

> I've seen some other discussions now on spam-fighting techniques with limited resources. But, along the lines of the discussion in the original post, comparing alternative choices for applications, is there particular mail software (or array of software) that is uniquely good on limited resources?

I'm also running a qmail setup and it seems minimal enough resource wise. I didn't post resource usage of various MTA's because I didn't have a spare box to setup non-qmail setups on to test with. Entirely without any proof I'd guess qmail is going to be on the lowend of resource usage compared to other MTAs because of the way it's designed and such, but like I said … no proof one way or other other. If anyone has a non-qmail setup and wouldn't mind posting some numbers we could compare and actually get some numbers (the above listings were done with a "ps vax").

> I'm not sure about greylisting, myself. It sounds like a great idea and I'm sure it works quite well, at least most of the time.

But it makes assumptions about the way email works that you really can't make. Why can't a re-send come from a different IP? Why can't there be a unique sender, if I'm running a list and want it that way?

I also looked at greylisting, but had similar concerns. One thing that particularly bothered me was the need to keep a whitelist up to date with a list of IPs running broken MTAs that won't try to rettempt delivery. That is more trouble that I want to go through for the little email traffic my linode gets, and for a higher traffic server I'd suspect you'd be running on hardware, or a linode, capable of running spamassassin.

I'm running exim on a Linode 120. It's also doing spamassassin and clamav, but that shouldn't affect the exim processes themselves.

  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
25072 ?        Ss     0:00     19   741  7882 2100  1.8 /usr/sbin/exim4 -bd -q30m
 2596 ?        S      0:00    138   741  7902 2888  2.5 /usr/sbin/exim4 -bd -q30m

top indicates that almost all of that memory is shared, so look at the bigger number rather than adding them up.

Here is a qmail setup:

PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
 1143 ?        S      0:00      1    31  1388  312  0.3 bin/qmail-inject -a -- xxx
 1417 ?        S      0:00      1    12  1371  308  0.3 supervise qmail-send
 1419 ?        S      0:00      1    12  1367  304  0.3 supervise qmail-smtpd
 1421 ?        S      0:00      1    12  1367  304  0.3 supervise qmail-pop3d
 1432 ?        S      0:00      0    18  1501  364  0.4 multilog t s100000 n20 /var/log/qmail/qmail-smtpd
 1434 ?        S      0:00      1    36  1523  484  0.5 qmail-send
 1435 ?        S      0:00      0    18  1501  364  0.4 multilog t s100000 n20 /var/log/qmail/qmail-send
 1436 ?        S      0:00      0    36  1547  540  0.6 /usr/local/bin/tcpserver -v -R -l xxx -x /etc/tcp.smtp.cdb -c 30 -u 1008 -g 1005 0 smtp rblsmtpd -r sbl-xbl.spamhaus.org -r bl.spamcop.net -r opm.blitzed.org /var/qmail/bin/qmail-smtpd xxx /home/vpopmail/bin/vchkpw /usr/bin/true
 1437 ?        S      0:00      1    36  1371  312  0.3 tcpserver -H -R -v -c100 0 110 qmail-popup xxx /home/vpopmail/bin/vchkpw qmail-pop3d Maildir
 1438 ?        S      0:02      0    18  1505  372  0.4 multilog t s100000 n20 /var/log/qmail/qmail-pop3d
 1443 ?        S      0:00      1    13  1503  348  0.4 qmail-lspawn ./Maildir
 1444 ?        S      0:00      1     9  1502  360  0.4 qmail-rspawn
 1445 ?        S      0:00      0     6  1501  344  0.4 qmail-clean

Looks pretty similar. We know that shared memory will throw all this off, but if we ignore it and just add up the resident sizes, yours and mine are only a couple hundred K off.

Of course it's hard to say just what difference the amount of system memory makes, too.

To be fair that qmail listing also includes pop3 and such, take those out of the equation since last time I checked exim didn't do pop3 and it's a bit larger of a gap … but still fairly negliable. Worth noting for the last little tweak, though.

Oh, you're right; I didn't notice that POP3 daemon there too. No, I'm using Courier for IMAP and POP. (Maildir on reiser rocks!)

That's on my list of things to do. I have courier running for imap as horde requires it, but I want to move the vpopmail folder over to reiserfs because of the small file benefits.

@Xan:

I'm not sure about greylisting, myself. It sounds like a great idea and I'm sure it works quite well, at least most of the time.

But it makes assumptions about the way email works that you really can't make. Why can't a re-send come from a different IP? Why can't there be a unique sender, if I'm running a list and want it that way?

No problem, at least with SQLgrey :
* it's smart enough to allow multiple IP in the same class C network to try to send the same email,

• when IPs are not from the same network, default whitelists are available from a central repository (hosted on a Linode 80 :) ) which means you benefit from other's whitelists if they post new entries on the user's mailing-list.

• it has provisions to handle most mailing-list unique sender schemes.

Here's a couple of test results comparing apache2 and lighttpd performance. The results shouldn't be all that surprising, but here they are anyway.

Relevent lighttpd conf bits:

fastcgi.server = ( ".php" => ((
        "bin-path" => "/usr/bin/php-cgi",
        "socket" => "/tmp/php.socket",
        "max-procs" => 2,
        "min-procs" => 2,
        "bin-environment" => (
                "PHP_FCGI_CHILDREN" => "1",
                "PHP_FCGI_MAX_REQUESTS" => "1000"
        ),
        "bin-copy-environment" => (
                "PATH", "SHELL", "USER"
        ),
        "broken-scriptfilename" => "enable"
)))

Relevent Apache2 conf bits:

StartServers 1
MinSpareServers 1
MaxSpareServers 3
ServerLimit 64
MaxClients 64
MaxRequestsPerChild 100

In both setups eAccelerator is used and identical url lists as well. The setup consists of http and https request to a half dozen vhosts.

Tests were conducted using http_load with these parameters:

./http_load -verbose -parallel 100 -seconds 20 urls

lighttpd: 12.4793 fetches/sec, 154712 bytes/sec

apache2: 0.018649 fetches/sec, 21.6142 bytes/sec

Lighttpd remained reached a max load of .42 while apache2 hit a 16.51 load. Additionally, lighttpd remained at a used about 20M ram (the majority by php-cgi processes) and did not need to resort to swap space at all. Apache2, on the other hand, chewed well into the swap space by over 150M. Finally, lighttpd remained responsive the entire time, while apache2 served only timeout messages and thrashed.

To further test lighttpd I ran the above http_load line from 3 different machines simultaneously to ensure the client was not being a bottleneck and obtained nearly identical results to that shown above.

Worth mentioning is that the node these test were run on is one of the beta xen nodes (80M ram).