How to accelerate Gentoo updates and reduce load on the host

Dilemma.

Hearing that a Gentoo user has joined your host is probably a lot like having a registered sex offender move into your neighborhood… well except the Gentoo guy isn't required to go around introducing himself, but you get the idea.

I have been a Gentoo user for many years and while I have my own dedicated machines to perform the updates, it's never really been a problem to do so on the machine recieving the update itself. CPU cycles, memory, and disk acces are all scarce resources on a linode host for the purposes of building software packages--even more so on the lesser expensive linode packages waves. There are a few things you can do to lessen the pain of regular software updates without having to compromise the source-based nature of Gentoo.

My goal for this howto is to help people achieve not only faster build times, but reduced load on their own virtual machine and the host machine also--your neighbors will appreciate it.

Leverage.

Like I was saying earlier CPU cycles, physical memory, and disk access are all scarce resources on a Linode. However, the available bandwidth and data transfer speeds are very nice. If you're anything like me (and I suspect many of you are), then you also have a much bigger bandwidth pool than you actually use each month. You will also probably have access to a linux machine with much better CPU and Memory. What I propose is you play to your strengths and use the proper tools for the proper application by performing software updates on your own machine (or whoever it may belong to :wink: ) and let your Linode do what it does best–run your services!

Execution.

The best way to use this abundance of bandwidth on the linode and your access to another beefy linux box to your advantage is to move the build process off the box entirely! All you have to do is duplicate parts of your Linode's filesystems onto your build machine. I don't mean that you should make an entire backup and then restore the system there and boot into it, that would be way too much work and entirely unnecessary. I mean you should copy things that would be affected by an emerge operation onto your build machine and then chroot into it much like in the Gentoo install instructions or in Faketoo. I use a script much like this one to make the backup tarball and scp it to my home pc.

It's as simple as:

buildbox# mkdir -p /mnt/linode
buildbox# tar -xjpvf linode-system.tar.bz2 -C /mnt/linode
(if your backup script is like mine, it doesn't back up portage, so you need to install a snapshot)
buildbox# tar -xjpvf (portage snapshot tarball) -C /mnt/linode/usr

buildbox# mkdir -p /mnt/linode/proc
buildbox# mount -t proc none /mnt/linode/proc
buildbox# cp -L /etc/resolv.conf /mnt/linode/etc/resolv.conf

buildbox# chroot /mnt/linode /bin/bash
linode# env-update
linode# source /etc/profile
linode# emerge sync
(now effectively in single user mode of your linode machine)

What you do from there is up to you, but I imagine you will want to:

linode# emerge (package name) -ba
-or-
linode# emerge world -uDab

It is important to use the -b (–buildpkg) option so that binary packages are built. When you are done building the packages you want to make a tarball of everything in $(PKGDIR), scp that bad boy to your Linode, extract the binary packages back to $(PKGDIR) and then merge the updated packages to your system like so:

linode# emerge sync && emerge world -uDka
-or-
linode# emerge sync && emerge (package name) -ka

Easy as pie!

****Edit:

Speeding up emerge sync on build machine****

Another speed improvement here is to have rsyncd running on the build box if you keep the build box portage updated. This will help a TON if you happen to use your build box to build packages for diverse systems. use: SYNC="rsync://localhost/gentoo-portage" emerge sync after chrooting into the build environment and it will update its portage tree from the host (buildbox) machine. Be careful if you're using rsync exclusions, that could cause problems.