Help Required! - Website expecting big increase in traffic

You could try something like httpperf or Apache JMeter or a commercial load tester and a test Linode to see whether it can handle it. If it can't then set up load balancing.

I don't know what stack you're using but if you're using nginx then the proxy/fastcgi cache can improve wordpress's performance vastly by caching the php pages for anonymous users and bypassing php all together.

Low-hanging fruit (short-term solutions):

1) put database on different linode

2) wordpress caching plugins

3) CloudFlare and associated plugin

4) Put static content on a separate server (or CDN)

Next step after that is to go properly multi-server, which is beyond the advice I can give, since it has a whole lot of other implications for wordpress. That said, you can get pretty damned far with two or three big beefy single-purpose linodes without getting into a multi-server type setup. Splitting off the database is easy, because there's still just one web server and one database server. Splitting the web server so you have multiple copies of the site is much harder.

Thanks very much for the responses all:

jebblue - this sounds like a great thing to do however given the current use often hits the the ceiling on the allowed 400% cpu useage it would suggest that if there is a 400% increase in traffic and users then this would not suffice. But could certainly be used to test a new configuration.

obs - Currently it is set up on Centos5.6 64Bit with Cpanel/WHM. I am not technically apt enough to understand the config you are suggesting but it sounds very manual and would require a lot of fiddling around and monitoring… but I may be wrong. My server manager would likely understand it more.

Guspaz - This is the sort of thing I was thinking of but need to ensure it can be done and not crash. We currently have the CloudFlare plugin with Cpanel but I was looking to go with someone like CacheFly so we can get a commission, but once again I have not used them before and do not know how to "Put static content on a separate server (or CDN)" as you say i.e. isolate the content. Also I notice you did not mention the use of the Linode load balancers if more than 1 server is used?

OVERALL

If a solution can be found I we could ask the client to pay for the extra help to sort it out.

Timing is pretty critical and would need this sorted by the end of the week. I like the idea of breaking the site up onto servers as it makes it more resilient than just shoving it into an elastic cloud.

PARTS TO THE SITE

At a guess the site could be broken into main site, forum, payment gateway? but I do not understand how to actually break it up.

I look forward to hear more!!

Cheers

400% increase doesn't mean anything if it's an increase that the server can handle. I think you're asking how to fix the server so it can handle it. Are you sure it's not handling it? If so then that is how you should have characterized your first post. If so then what is failing? You should set up a side Linode a clone and hit it with httpperf (et al, etc. etc.) all day on and off cycle until you know what the problem is. Then fix the problem.

Hi Jebblue,

thanks for your response… sorry if I am being unclear.

There is nothing wrong with the server at present. However, starting next week the owners are going through a very intensive marketing campaign to attract new users and given the response from previous marketing campaigns they are expecting to have what is currently 40K registered users increase to 100-160K registered users, and from 200 concurrent users to 35K concurrent users.

Currently the CPU hangs around 150% but when they market it goes to 300% + and occasionally hits the full 400% cpu. The marketing campaign planned for next week is significantly larger, than previously done. Also the size of the files that new clients download is increasing from approx 3-4mb to 12mb.

So… I want to be prepared

Just to mention, I need to sort this out by the end of this week.

I would hate to have to pack up and move simply because I could not get a solid design support/consultant using Linode.

Is there a job posting section to this forum to speed this up?

Thanks in advance

@OnthegOinOz:

I would hate to have to pack up and move simply because I could not get a solid design support/consultant using Linode.
Yeah, that whole Linode is a UNMANAGED VPS service is oh so confusing.

Yeah I get the unmanaged part, but it works! its been solid! Surely there are people that offer a service to help people use it and configure their sites/services????

There are, but this here is a community support forum, where people help out in their spare time, so it's not necessarily the greatest venue for time critical things.

In terms of my not mentioning Linode's load balancing, I didn't for two reasons:

1) It would only be required if you had multiple front-end servers, which was the "much harder" part that I mentioned, one that I have no experience with

2) They have a maximum concurrent user limit of 10k, if memory serves, which sounds like it may not be well suited for your uses. There are also other techniques, such as DNS round robin, or a simple redirect (a linode sits on www.foo.com and redirects users to www1.foo.com, www2.foo.com, etc), or a reverse proxy like Varnish, etc.

One good thing to take a look at immediately might be, where is that current CPU use coming from? As in, if you log into the box and run top, what processes are using the CPU time? What's the box's disk IO like? What does the memory usage (in terms of total free/cached, total in use, total swapped, what processes are using how much, etc) look like?

@Guspaz:

2) They have a maximum concurrent user limit of 10k, if memory serves, which sounds like it may not be well suited for your uses. There are also other techniques, such as DNS round robin, or a simple redirect (a linode sits on http://www.foo.com and redirects users to www1.foo.com, www2.foo.com, etc), or a reverse proxy like Varnish, etc.
They have a maximum concurrent connection limit of 10k. If connections take less than a second – and I believe NodeBalancers disable HTTP keep-alive -- you can fit in a lot more. Or you can get more than one NB and DNS round-robin them.