Magento and Asterisk on same VPS - really really bad?

I'm running Magento CE 1.9.1.0 on a Linode 6GB account ($60/mo, 6 GB RAM, 4 CPU cores) on Debian 7.6 64-bit. It may be an overkill already as there's max 20-concurrent visitors to the site, but I'm reluctant to drop it to a Linode 4GB account.

Now I'm getting into VoIP so I wanted to set up Asterisk 13 to do some basic IVR/call recording/extensions etc. There'll be max 3 concurrent voice calls for the next few months, and realistically probably max 10 concurrent calls in the foreseeable future. I won't be using any massively taxing features such as speech recognition or text-to-speech…in the beginning at least. ;) And I won't hesitate to upgrade to a Linode 8GB account if necessary to speed things up. However…

Is it even possible to run Magento and Asterisk on the same VPS? I'm asking because of this warning:

"Please note: Because of the special configuration options required for this setup, you should not run other services on the Linode you intend to use Asterisk on. It is also worth noting that this guide will walk you through using PV-GRUB." (Deploy VoIP Services with Asterisk and FreePBX on Ubuntu 12.04 (Precise))

If it's really not possible, do you think I can get decent performance running Asterisk on a separate $20 Linode 2GB account? Or should I do 2X Linode 4GB accounts?

What do you guys think?

Thanks!