Ran LoadImpact on site and got terrible results

I ran a load impact on a Linode 512 site of mine last night and the results were not good at all. The test graph shows my load time being a second and under for 11 SBU* (or Simulated Browser Users as they call it see the bottom definition), but then the load time spikes between 1 and 10 seconds after that. Then the load time stays consistently above 5 seconds after 20 concurrent users climbing to 1.4 minutes at 50 concurrent users (this is certainly how it felt when I was using it simultaneously)!

While I ran this test, I had htop running so I could see how my resources were doing. The thing I noticed though is that the memory usage never climbed above 112Mb and it typically idles around 83 mb when not being tested.

My site uses php, mysql, apache, and javascript a lot and the homepage being tested by load impact does not require accessing my MySQL database if that is relevant. In fact the page I point load impact at is all static content. The php that runs on the tested page is to include other php files like the header and footer as well as run a switch statement to check the session vars to see if the user is logged in (which loadimpact is not of course) and decide to load a static php file since load impact will not be logged in.

I have done all the optimizations suggested in the getting started guide for a linode 512 (like setting maxclients to 24).I also set an expiration time of a day in apache for css, jpg, js, png, files. I also set apache to deflate css and js files. Beyond this, I have not done anything like install php cachers or optimizers.

I know this was a pretty long post but does anyone have any clues? I would greatly appreciate the help.

*Load Impact defines SBU in the following way:

Then there is the Simulated Browser User (SBU). This user type can use multiple concurrent network connections when loading resources from a single target host. This means that if e.g. the simulated user needs to fetch four different resources from the same target host, it can open four concurrent network connections and transfer all four resources in parallel. Obviously, this results in faster page loads, but also a lot more stress on the target server that has to serve four connections at once instead of just one.