DRBD how-to?
I'm kinda new to Linode - no doubt part of the problem
I wonder if anyone knows of one, or perhaps might like to contribute to one? Or maybe just a hint or nudge in the right direction?
Many thanks!!
Rob
12 Replies
Best send through the errors though; i've compiled earlier versions of drbd on Etch before and might be able to help.
I'm at a clean distribution, no modifications yet - first boot for testing. It's Debian 5.0.
li34-206:~# uname -a
Linux li34-206 2.6.26-linode13 #1 SMP Sun Jul 13 19:11:25 EDT 2008 i686 GNU/Linux
As you suggested, I tried drbd8-modules:
apt-get install drbd8-modules-2.6-686
When it gets into installing, it wants kernel source (or at least headers) and tries to get them with apt-get. I've tried this both ways - stopping and grabbing source from
The following NEW packages will be installed:
busybox drbd8-modules-2.6-686 drbd8-modules-2.6.26-1-686 initramfs-tools klibc-utils libc6-i686 libklibc libvolume-id0 linux-image-2.6-686
linux-image-2.6.26-1-686 udev
Along this path, apt-get wants to set up initrd.imgs, initramfs, a system map, another kernel config file, etc. /boot gets populated, though my understanding is a Linode VPS won't be using that.
BUT, it does create files under /lib/modules:
-rw-r--r-- 1 root root 239470 Dec 20 11:18 2.6.26-1-686/extra/drbd8/drbd/drbd.ko
(I don't know why it's dated Dec 20.)
I'm a rusty FreeBSD and OpenBSD kinda guy, so I don't understand some of the Linux kernel-related programs I'm running. But I figured since modprobe -l doesn't come up with anything, I could try moving the 'extra' folder from the other 2.6.26 and doing a depmod.
So finally I get the error you request, which probably has more to do with my process than anything else
li34-206:/lib/modules# modprobe -l
/lib/modules/2.6.26-linode13/extra/drbd8/drbd/drbd.ko
li34-206:/lib/modules# modprobe drbd
FATAL: Error inserting drbd (/lib/modules/2.6.26-linode13/extra/drbd8/drbd/drbd.ko): Invalid module format
First, I'm downloading the kernel source for the kernel I'm running, getting bzip2, and untar'ing the kernel. (I don't think there's anything interesting in the output from most of these commands, so I'm just listing what I typed, not what comes back.)
# cd /usr/src/
# uname -a
Linux li34-206 2.6.18.8-linode16 #1 SMP Mon Jan 12 09:50:18 EST 2009 i686 GNU/Linux
# wget http://www.linode.com/src/2.6.18.8-linode16.tar.bz2
# apt-get install bzip2
# tar xvjf 2.6.18.8-linode16.tar.bz2
From there, I'm using the writeup at
Made a symlink to linux:
# ln -s 2.6.18.8-linode16 linux
I don't actually have tools yet, so next was:
# apt-get install build-essential
Next is to prep the kernel source apparently:
# make mrproper
CLEAN scripts/basic
CLEAN scripts/genksyms
CLEAN scripts/kconfig
CLEAN scripts/mod
CLEAN scripts
CLEAN include/config
CLEAN .config .config.old include/asm .version include/linux/autoconf.h include/linux/version.h include/linux/utsrelease.h Module.symvers
I don't have the DRBD8 source yet, so I get that from
# cd /usr/src/
# wget http://oss.linbit.com/drbd/8.3/drbd-8.3.1.tar.gz
# tar xvzf drbd-8.3.1.tar.gz
From there I should be able to do a make clean all, but it doesn't quite work out that way. After I apt-get install flex and try a make clean; make all, I start to run into issues:
# make all
make -C drbd drbd_buildtag.c
make[1]: Entering directory `/usr/src/drbd-8.3.1/drbd'
...
make[1]: Leaving directory `/usr/src/drbd-8.3.1/documentation'
Userland tools build was successful.
Could not determine uts_release
make: *** [module] Error 1
I don't know why it would choke on determining uts_release (or really what it's doing, other than trying to identify the kernel), but I believe I should be able to work around it by specifying the kernel version and source directory, as follows. I'm pasting all the results in, though, since it doesn't actually work…
# make KVER=2.6.18.8, KDIR=/usr/src/linux all
make -C drbd drbd_buildtag.c
make[1]: Entering directory `/usr/src/drbd-8.3.1/drbd'
make[1]: Leaving directory `/usr/src/drbd-8.3.1/drbd'
make[1]: Entering directory `/usr/src/drbd-8.3.1/user'
cp ../drbd/drbd_buildtag.c drbd_buildtag.c
gcc -g -O2 -c -W -Wall -I../drbd -c -o drbd_buildtag.o drbd_buildtag.c
gcc -o drbdadm drbdadm_scanner.o drbdadm_parser.o drbdadm_main.o drbdadm_adjust.o drbdtool_common.o drbdadm_usage_cnt.o drbd_buildtag.o drbdadm_minor_table.o
gcc -o drbdmeta drbdmeta.o drbdmeta_scanner.o drbdtool_common.o drbd_buildtag.o
gcc -o drbdsetup drbdsetup.o drbdtool_common.o drbd_buildtag.o drbd_strings.o
make[1]: Leaving directory `/usr/src/drbd-8.3.1/user'
make[1]: Entering directory `/usr/src/drbd-8.3.1/scripts'
Makefile:49: No special distribution INITD
Makefile:50: setting INITD=/etc/init.d/ according to LSB
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/usr/src/drbd-8.3.1/scripts'
make[1]: Entering directory `/usr/src/drbd-8.3.1/documentation'
To (re)make the documentation: make doc
make[1]: Leaving directory `/usr/src/drbd-8.3.1/documentation'
Userland tools build was successful.
make[1]: Entering directory `/usr/src/drbd-8.3.1/drbd'
Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/usr/src/linux
test -f ../scripts/adjust_drbd_config_h.sh && \
KDIR=/usr/src/linux O= /bin/bash ../scripts/adjust_drbd_config_h.sh
/usr/src/linux /usr/src/drbd-8.3.1/drbd
ls: cannot access /usr/src/linux/include/linux/version.h: No such file or directory
Sorry, automagic adjustment of drbd_config.h failed.
For well known 2.6\. kernels, no adjustment to the shipped drbd_config is necessary.
You need to verify it yourself.
make[1]: [kbuild] Error 2 (ignored)
make -C /usr/src/linux SUBDIRS=/usr/src/drbd-8.3.1/drbd modules
make[2]: Entering directory `/usr/src/2.6.18.8-linode16'
ERROR: Kernel configuration is invalid.
include/linux/autoconf.h or include/config/auto.conf are missing.
Run 'make oldconfig && make prepare' on kernel src to fix it.
WARNING: Symbol version dump /usr/src/2.6.18.8-linode16/Module.symvers
is missing; modules will have no dependencies and modversions.
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_buildtag.o
cc1: error: include/linux/autoconf.h: No such file or directory
make[3]: *** [/usr/src/drbd-8.3.1/drbd/drbd_buildtag.o] Error 1
make[2]: *** [_module_/usr/src/drbd-8.3.1/drbd] Error 2
make[2]: Leaving directory `/usr/src/2.6.18.8-linode16'
make[1]: *** [kbuild] Error 2
make[1]: Leaving directory `/usr/src/drbd-8.3.1/drbd'
So it told me to run "make oldconfig && make prepare", so I did those (in two commands rather than with the &&, but I don't see any errors.) Then I go back into the drbd8 source tree and try again:
li34-206:/usr/src/drbd-8.3.1# make KVER=2.6.18.8, KDIR=/usr/src/linux all
make -C drbd drbd_buildtag.c
make[1]: Entering directory `/usr/src/drbd-8.3.1/drbd'
...
make[1]: Leaving directory `/usr/src/drbd-8.3.1/documentation'
Userland tools build was successful.
make[1]: Entering directory `/usr/src/drbd-8.3.1/drbd'
Calling toplevel makefile of kernel source tree, which I believe is in
KDIR=/usr/src/linux
test -f ../scripts/adjust_drbd_config_h.sh && \
KDIR=/usr/src/linux O= /bin/bash ../scripts/adjust_drbd_config_h.sh
/usr/src/linux /usr/src/drbd-8.3.1/drbd
/usr/src/drbd-8.3.1/drbd
Adjusted drbd_config.h:
--- ./linux/drbd_config.h 2009-03-27 11:17:55.000000000 +0000
+++ ./linux/drbd_config.h.new 2009-04-01 16:56:28.000000000 +0000
@@ -90 +90 @@
-//#define HAVE_LINUX_BYTEORDER_SWABB_H
+#define HAVE_LINUX_BYTEORDER_SWABB_H
make -C /usr/src/linux SUBDIRS=/usr/src/drbd-8.3.1/drbd modules
make[2]: Entering directory `/usr/src/2.6.18.8-linode16'
WARNING: Symbol version dump /usr/src/2.6.18.8-linode16/Module.symvers
is missing; modules will have no dependencies and modversions.
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_buildtag.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_bitmap.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_proc.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_worker.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_receiver.o
/usr/src/drbd-8.3.1/drbd/drbd_receiver.c: In function 'drbd_crypto_alloc_digest_safe':
/usr/src/drbd-8.3.1/drbd/drbd_receiver.c:2709: warning: passing argument 1 of 'crypto_alloc_hash' discards qualifiers from pointer target type
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_req.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_actlog.o
CC [M] /usr/src/drbd-8.3.1/drbd/lru_cache.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_main.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_strings.o
CC [M] /usr/src/drbd-8.3.1/drbd/drbd_nl.o
LD [M] /usr/src/drbd-8.3.1/drbd/drbd.o
Building modules, stage 2.
MODPOST
/bin/sh: scripts/mod/modpost: No such file or directory
make[3]: *** [__modpost] Error 127
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/usr/src/2.6.18.8-linode16'
make[1]: *** [kbuild] Error 2
make[1]: Leaving directory `/usr/src/drbd-8.3.1/drbd'
make: *** [module] Error 2
I'm sure much of that output isn't relevant, but I only clipped the userland stuff.
So now I'm looking for modpost, I guess. I found some stuff online suggesting that I might need to do a make-kpkg kernel-image to get modpost, so I started with apt-get install kernel-package to get make-kpkg, then a make-kpg kernel-image.
Naturally, there's a lot of output here, but I'll paste from just before the word "FATAL" to the bottom…
make[2]: Entering directory `/usr/src/2.6.18.8-linode16'
if [ -r System.map -a -x /sbin/depmod ]; then /sbin/depmod -ae -F System.map -b /usr/src/linux/debian/linux-image-2.6.18.8-linode16 -r 2.6.18.8-linode16; fi
make[2]: Leaving directory `/usr/src/2.6.18.8-linode16'
test ! -e scripts/package/builddeb || mv -f scripts/package/builddeb scripts/package/builddeb.kpkg-dist
test ! -e scripts/package/Makefile || test -f scripts/package/Makefile.kpkg-dist || (mv -f scripts/package/Makefile scripts/package/Makefile.kpkg-dist && (echo "# Dummy file "; echo "help:") > scripts/package/Makefile)
test ! -f Documentation/lguest/lguest || \
install -p -o root -g root -m 644 Documentation/lguest/lguest /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/lguest
test ! -f /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/lguest || \
chmod 755 /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/lguest
test ! -e /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/source || \
mv /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/source ./debian/source-link
test ! -e /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/build || \
mv /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/build ./debian/build-link
test ! -e ./debian/source-link || \
mv ./debian/source-link /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/source
test ! -e ./debian/build-link || \
mv ./debian/build-link /usr/src/linux/debian/linux-image-2.6.18.8-linode16/lib/modules/2.6.18.8-linode16/build
/sbin/depmod -q -FSystem.map -b /usr/src/linux/debian/linux-image-2.6.18.8-linode16 2.6.18.8-linode16;
FATAL: Could not open 'System.map': No such file or directory
make[1]: [debian/stamp/install/linux-image-2.6.18.8-linode16] Error 1 (ignored)
test ! -f System.map || cp System.map \
/usr/src/linux/debian/linux-image-2.6.18.8-linode16//boot/System.map-2.6.18.8-linode16;
test ! -f System.map || chmod 644 \
/usr/src/linux/debian/linux-image-2.6.18.8-linode16//boot/System.map-2.6.18.8-linode16;
cp arch/i386/boot/bzImage /usr/src/linux/debian/linux-image-2.6.18.8-linode16//boot/vmlinuz-2.6.18.8-linode16
cp: cannot stat `arch/i386/boot/bzImage': No such file or directory
make[1]: *** [debian/stamp/install/linux-image-2.6.18.8-linode16] Error 1
make[1]: Leaving directory `/usr/src/2.6.18.8-linode16'
make: *** [kernel-image] Error 2
I may be defeatist, but it's usually about this time I wonder if I made a wrong turn somewhere back in Kansas and maybe that's why I'm in the desert rather than New York…
At any rate, I've found references online to this problem suggesting "You need to do this as root" (I am) and "Just curious, why not use the packages provided for this?" (see previous post). Alas, I don't see an answer for me…
I also saw a non-native English speaker preface a code section on a mailing list with the poetic statement, "This is a recapitulation to my manipulation," so although I haven't solved this issue, I pause again with a smile!
So in my previous post, I tried to use the prepackaged module and dropped the ball. In this one, I'm trying to compile from source. I know there's a way to do it with module-assistant, too, and I can write it up, but as I recall it craps out when trying to build drbd8-module-source. When getting the source, it's redirected to drbd8-source (which seems to be correct) and it downloads fine. But then it still tries to build drbd8-module-source, which it doesn't actually have, so of course it fails.
Hints, tips, tricks, direction - all welcome and I'm very grateful for the help.
Thanks!
Rob Martin
Unfortunately it's a bit beyond me on how to fix these compilation errors
I did note that this howto
For what it's worth it looks like you're on the right track, resolving that MODPOST error does seem to involve make-kpkg kernel-image.. what a pain though..
One thing I did notice, in that last lot of errors:
cp: cannot stat `arch/i386/boot/bzImage': No such file or directory
That path looks like it expects to be relative to the directory you're in - I suspect it should be inside the /usr/src/2.6.18.8-linode16 - is make smart enough to handle that since it 'enters' the directory at the beginning of the script or is it expecting you to be inside that directory when you run make? But you probably were already.
Sorry, I'm kind of out of ideas otherwise
Maybe I'll try pasting the forum link into a support ticket and see what happens. It's worth a try
First, we need a few packages that aren't in our stock image:
apt-get install build-essential flex bison wget unifdef
Next, I grabbed the kernel sources and built them:
cd /usr/src
wget http://www.linode.com/src/2.6.18.8-linode16.tar.bz2
tar -jxf 2.6.18.8-linode16.tar.bz2
cd 2.6.18.8-linode16
make -j4
There is a bunch of output (including a few warnings you can ignore) but the final output should be:
Kernel: arch/i386/boot/vmlinuz is ready (#2)
Assuming the kernel builds correctly, install headers and make the /usr/src/linux link:
make headers_install
cd ..
ln -s 2.6.18.8-linode16 linux
Next, I grabbed the drbd sources and unpacked them (still in /usr/src/ but that shouldn't matter) and built them:
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.1.tar.gz
tar -zxf drbd-8.3.1.tar.gz
cd drbd-8.3.1
make KDIR=/usr/src/linux/
Running make with -j4 here seemed to break it, but it's not a big module so the compile time difference is minimal. When it finishes the last bit of output should be:
Module build was successful.
If you get this far and everything has worked, you should be golden. Just install the module and modprobe it:
make install
modprobe drbd
lsmod
If everything went according to plan, lsmod's ouput should be similar to this:
Module Size Used by
drbd 257688 0
If not, well, post errors and maybe someone can spot the problem.
Note: It seems to me that a failed build breaks something in the directory that causes all subsequent builds to fail. Without the KDIR passed to make the build failed (that's where the uts_release error is coming from) because it couldn't find header files. After I figured it out there was a different build error that I finally got around by deleting the directory and unpacking the source again. Really annoying, but an easy fix. My advice is to blow away the source directory you have and start from a clean one.
Also, I did this build on an Ubuntu 8.04 Linode because I didn't have a clean Debian Lenny one ready to roll. I'll set one up tomorrow and test there if you have troubles, but in theory it should work on Lenny without issues.
Hopefully that helps.
-James
So I'm starting with a fresh Debian 5.0 (Lenny) kernel, making a new Linux distribution in the Linode Manager and using the "Latest 2.6 Series" kernel, 2.6.18.8-lnode16. I have /dev/xvdh pointed to a disk image I've created and copied between two linodes.
I'm installing packages needed for kernel work (per irgeek's recommendation):
apt-get install build-essential flex bison wget unifdef
Then retrieving kernel source, untar'ing it, building the kernel, making the headers, and creating the symlink:
cd /usr/src
wget http://www.linode.com/src/2.6.18.8-linode16.tar.bz2
tar -jxf 2.6.18.8-linode16.tar.bz2
cd 2.6.18.8-linode16
make -j4
make headers_install
cd ..
ln -s 2.6.18.8-linode16 linux
Then grabbing the DRBD8 sources, untar'ing them, and building the module:
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.1.tar.gz
tar -zxf drbd-8.3.1.tar.gz
cd drbd-8.3.1
make KDIR=/usr/src/linux/
make install
(Can I just say thank you again, irgeek? Everything has worked perfectly in Lenny exactly as you described.)
And then… modprobe fails.
# modprobe drbd
FATAL: Error inserting drbd (/lib/modules/2.6.18.8-linode16/kernel/drivers/block/drbd.ko): Invalid module format
My understanding is that an "invalid module format" often means you're not building the module against the kernel you're actually running. I suppose this is possible, but I believe I've grabbed the correct sources and they match the selection I've made in creating my linode's kernel distribution.
However, I've seen something online suggesting that even the "extraversion" information on the kernel can cause this, and that perhaps I need to "build the kernel the Debian way" using make-kpkg with "–append-to-version -linode16".
So to try this, then, I'm running an apt-get for tools used in the Debian way of kernel building. This actually installs 24 packages.
apt-get install kernel-package libncurses5-dev
Then I go back to the kernel source directory and try building again. I'm skipping kernel configuration and jumping right to the make-kpg. My understanding is that make-kpkg basically does "make dep; make clean; make bzImage; make modules" but I don't know where the "–append-to-version" switch gets placed, so I'm using the whole package.
make-kpkg --append-to-version=-linode16
It was quick, painless, and ran without errors. So now I'll go back to drbd and try again…
cd /usr/src/drbd-8.3.1
make KDIR=/usr/src/linux/
make install
modprobe drbd
And I still get an invalid module format.
So I start poking around, and try a modinfo on the new drbd.ko file:
# modinfo /lib/modules/2.6.18.8-linode16/kernel/drivers/block/drbd.ko
filename: /lib/modules/2.6.18.8-linode16/kernel/drivers/block/drbd.ko
alias: block-major-147-*
license: GPL
description: drbd - Distributed Replicated Block Device v8.3.1
author: Philipp Reisner <phil@linbit.com>, Lars Ellenberg <lars@linbit.com>depends:
vermagic: 2.6.18.8-linode16 SMP mod_unload Xen PENTIUM4 REGPARM gcc-4.3
parm: minor_count:Maximum number of drbd devices (1-255) (uint)
parm: allow_oos:DONT USE! (bool)
parm: cn_idx:uint
parm: enable_faults:int
parm: fault_rate:int
parm: fault_count:int
parm: fault_devs:int
parm: trace_level:int
parm: trace_type:int
parm: trace_devs:int
parm: proc_details:int
parm: usermode_helper:string</lars@linbit.com></phil@linbit.com>
and double check my gcc:
# gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.2-1.1' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-cld --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.3.2 (Debian 4.3.2-1.1)
I notice that the architecture is set to i486-linux-gnu, rather than i686-linux-gnu. But I want to check the kernel too:
# cd /usr/src/linux/scripts/
# chmod +x ver_linux
# ./ver_linux
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
Linux li34-206 2.6.18.8-linode16 #1 SMP Mon Jan 12 09:50:18 EST 2009 i686 GNU/Linux
Gnu C 4.3.2-1.1)
Gnu make 3.81
binutils Binutils
util-linux 2.13.1.1
mount 2.13.1.1
module-init-tools 3.4
e2fsprogs 1.41.3
Linux C Library > libc.2.7
Dynamic linker (ldd) 2.7
Procps 3.2.7
Net-tools 1.60
Kbd 83:
Sh-utils 6.10
Modules Loaded
So I think the gcc I've got is fine, and dpkg -l make tells me I'm at version 3.81-5. Aside from the target architecture being i486 (haven't got a clue if this is relevant!) I think the system's okay…
And then I think I find it!
# dmesg | tail -1
drbd: version magic '2.6.18.8-linode16 SMP mod_unload Xen PENTIUM4 REGPARM gcc-4.3' should be '2.6.18.8-linode16 SMP mod_unload Xen PENTIUM4 REGPARM gcc-4.2'
Bingo!
# apt-get install gcc-4.2
# rm /usr/bin/gcc
# ln -s /usr/bin/gcc-4.2 /usr/bin/gcc
# cd /usr/src/drbd-8.3.1
# make KDIR=/usr/src/linux/
# make install
# modprobe drbd
# lsmod
Module Size Used by
drbd 257688 0
# rm /usr/bin/gcc
# ln -s /usr/bin/gcc-4.3 /usr/bin/gcc
Okay, I think that works. I'm rebooting and will check it:
# lsmod
Module Size Used by
# modprobe drbd
# lsmod
Module Size Used by
drbd 257688 0
I'm calling that good.
So finally it comes down to a question. Is it a bug if the kernel we're using wasn't compiled with the same gcc version installed in the distribution? I understand this could be a problem - Debian Lenny has a newer version off gcc than Etch (at least I presume so) and the Ubuntu 8.04 distro (again, I presume) that irgeek used. So it's probably not a bug so much as an experience I've earned in exchange for virtually all of my labor this week
As a final note, I'll type this up under the Linux Tips, Tricks, and Tutorials section, using irgeek's instructions for a standard install (at least on Ubuntu 8.04) and my revised instructions on Debian 5.0.
EDIT: What spam?
@Stever:
How's a guy supposed to find his safe viagra online when people keep deleting those helpful links?
Google it.
On second thought… you better not.
@pclissold:
Well, there's spam, egg, sausage and spam; that's not got much spam in it.
EDIT: What spam?
;)
Who's been editing my posts?