Kernel issue or confused user? :_)

I have two close-to-identically configured VPS systems; a linode and a competitor using xen. The main difference is that I'm using my linode as my primary site for web and smtp access.

However; twice in the past month my linode has died with out-of-memory errors…

Kernel 2.6.23.1-linode36 on an i686

Out of memory: kill process 29571 (index.panix-pro) score 143 or a child
Killed process 29649 (find)
Out of memory: kill process 26924 (swish-e) score 132 or a child
lowmem_reserve[]: 0 0
Normal: 3134*4kB 98*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*
2048kB 0*4096kB = 13928kB
Swap cache: add 38174864, delete 38174824, find 10480667/18191565, race 31+1313
Free swap  = 240976kB
Total swap = 263160kB

OK… so how can I be out of memory when I have 2.4Gb of free swap available?

Killed process 27446 (proxymap)
Out of memory: kill process 27746 (index.cgi) score 999 or a child
Killed process 27746 (index.cgi)
Out of memory: kill process 27771 (index.cgi) score 999 or a child
Killed process 27771 (index.cgi)
Out of memory: kill process 27779 (index.cgi) score 910 or a child
Killed process 27779 (index.cgi)
Out of memory: kill process 23339 (verify) score 684 or a child
Killed process 23339 (verify)
printk: 137 messages suppressed.

Yeah well, once the kernel is confused…

autoblock invoked oom-killer: gfp_mask=0xd0, order=2, oomkilladj=0
0976fcb8:  [<08069652>] dump_stack+0x22/0x30
0976fcd0:  [<080b374d>] out_of_memory+0x11d/0x130
0976fd04:  [<080b4a2e>] __alloc_pages+0x28e/0x330
0976fd54:  [<080b4afc>] __get_free_pages+0x2c/0x50
0976fd60:  [<08087fa6>] dup_task_struct+0x36/0xc0
0976fd7c:  [<08088cd5>] copy_process+0x85/0xe00
0976fe4c:  [<08089ae4>] do_fork+0x74/0x210
0976feb0:  [<08080c42>] sys_clone+0x72/0x90
0976fee0:  [<0806ba03>] handle_syscall+0xd3/0xe0
0976ff50:  [<0807d2da>] handle_trap+0x2a/0x150
0976ff78:  [<0807da16>] userspace+0x1d6/0x250
0976ffd4:  [<0806b6be>] fork_handler+0x9e/0xb0
0976fffc:  [<00000000>] _stext+0xf7fb7f4c/0xc

The kernel is crashing???

Mem-info:
Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  20   Cold: hi:   62, btch:  15 usd:  55
Active:31 inactive:452 dirty:0 writeback:82 unstable:0
free:2769 slab:83569 mapped:1 pagetables:514 bounce:0
Normal free:11076kB min:2416kB low:3020kB high:3624kB active:124kB inactive:1808
kB present:365760kB pages_scanned:3301 all_unreclaimable? yes
lowmem_reserve[]: 0 0
Normal: 2457*4kB 80*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*
2048kB 0*4096kB = 11076kB
Swap cache: add 38171500, delete 38171412, find 10480526/18190665, race 31+1313
Free swap  = 170904kB
Total swap = 263160kB
Free swap:       170904kB
92160 pages of RAM
0 pages of HIGHMEM
3158 reserved pages
93 pages shared
88 pages swap cached

Still plenty of swap!

So… what's going on?