cpu over 400% uses for 2 hours, sites shutdown

Your Linode was OOMing. Take a look at your console log via Lish:

http://www.linode.com/wiki/index.php/Lish_Documentation

-Tom

Sorry to be such a noob, what do you mean by OOming, looking on google does not really help me.

Is there a way my server reboot itself at 100% of the cpu?

He means your server ran out of memory (Out Of Memory).

You need to do one or all of the following:

1) Reconfigure your services to consume less memory

2) Switch to more memory efficient services (IE, Apache -> Lighttpd/Lightspeed/nginx/etc)

3) Increase the amount of swap space available to your node

That's my take on it, anyhow.

EDIT: Under no circumstances should you ever want a computer to reboot when it consumes 100% of the available CPU time. Because that happens extremely frequently, and is normal. Your server would do nothing but reboot all day long.

I think I got it down for a while, it non stop ooming now…, and with all the links incoming I can't probably do nothing for all the day

pretty sad..

ok thx for the advice,

I added 180 meg of ram for the day, and activate super cache for wordpress. I think mysql was killing my server.

If you're able to log in at all (and you might be more successful with Lish at that), you should probably just shut down your web server entirely and then work on optimizing your box.

What web server do you use? Any dynamic languages enabled (PHP/Perl/etc)? Do you use MySQL? Any mail services (SpamAssasin is a memory hog)? Anything else running on the box? What Linode do you have?

MySQL can be configured for various memory levels. It ships with a bunch of pre-configured config files for various memory levels. You can try swapping out your config file with one of the lower memory ones.

I'm not very familiar with WordPress, although switching out Apache for Lighttpd is probably still sound advice; unless you're using a very custom Apache config, it's usually easier to install Lighttpd than tweak Apache for lower memory usage.

Do you have enough swap space on your node? You should probably have at least as much swap as you have RAM.

can you point my to witch config files I need to change, i just saw in lish that mysql was out of memory, tahts the problem

mysql was out of memory, thats the problem, it told me in lish, so i'm very popular of some bad people overwhelming my slq db

The example config files can be found here:

/usr/share/doc/mysql-server-5.0/examples

Most of them are gzipped so you'll need to decompress them. Then you can override your existing my.cnf (which is in /etc/mysql I think).

The headers of the files say what they're good for, but here's an online list:

http://brian.moonspot.net/2008/05/06/ex … cnf-files/">http://brian.moonspot.net/2008/05/06/example-mycnf-files/

I guess you probably want my-medium.cnf

Just because it was MySQL that ran out of memory doesn't mean that MySQL was the process that was using the most memory. If Apache used up all the RAM and then MySQL tried to allocate some more, MySQL would run out of RAM even though Apache was using most of it.

thank you for the advice!

I'm still a bit shaken, It took me 4 hours to get it back online, I updated the my.cnf config, and it did took the load for 2 hours and after the load went to lower levels.

This is really unfortunate as I make revenue from ads.

I know nothing about server optimization, before today I was happy it was up and running since my disk storage update,

I added 90 meg of memory permanently, i'm on a 360, earlier you talked about the swap, is this the swap image in the linode control you talked about? or i'm wrong?

@Karnius:

I added 90 meg of memory permanently, i'm on a 360, earlier you talked about the swap, is this the swap image in the linode control you talked about? or i'm wrong?

Yes, it is it. Have a look in the menu under disk images:

xxx MB Swap Image (swap)

You can resize (increase) the swap from this menu

Generally speaking, you do NOT want a large swap partition – if you're thrashing swap, you're already on the way down and are just delaying the inevitable by a few minutes. Swap is extremely slow and is no substitute for either reducing your memory requirements or increasing available memory.

I usually run with either 128MB or 256MB, depending on how much "inactive" stuff I expect. If it's a single-purpose server, 128MB is more than enough.

okay ..

My server was more stable this morning, but still crashed just now..

This is only one site, with like 5000 visit a day, I do not understand how this is crashing a dedicated.

I augmented the swap, my-medium,cfn is up and running, my last option is returning to managed hosting I think…..

Search the forums, there is plenty of advice on how to set up your server so it doesn't run out of memory.

this one for example: http://www.linode.com/forums/viewtopic.php?t=4096

It's usually the same problems. People run a million instances of apache and a memory-hogging mysql installation and are surprised when the box runs out of memory and starts swaping.

A few items you can do:

• reduce the number of apache processes

• reduce the mysql foot print (maybe try small ?)

• install munin to keep track of what's going on

• check with ps aux what processes are consuming all the memory

Maybe this should go into the FAQ or a tutorial as the problem comes up about once a month.

@Karnius:

okay ..

My server was more stable this morning, but still crashed just now..

This is only one site, with like 5000 visit a day, I do not understand how this is crashing a dedicated.

I augmented the swap, my-medium,cfn is up and running, my last option is returning to managed hosting I think…..

We can't really help you unless you give us more information. Post snapshots from ps and/or top right before the system crashes, for example.

hey,

Okay, I'm gonna post them tomorrow.

Thank you for all your help by the way.

For now I followed some advice from you guys:

Disabled innodb in my.cnf

Changed de maxclient at 35 in httpd.conf

Disabled spam filter in ispconfig

deleted red5 flash server communication

Put my swap to 450 meg

Looking into :

• postfix antivirus, i'm not sure where to disable this

• MinSpareServers, MaxSpareServers in httpd.conf were already at 1 ???

• wordpress mysql leaking ??

I run on a 540 meg today, and it seem stable since the last crash but I did not have huge spike like yesterday.

Hi,

I don't know what kind of configurations you are using, but I run a forum in a 360 linode with 2/3 or you traffic without any problems:

Here you can see my MySQL configuration:

http://www.vbulletin.com/forum/showthre … ost1696100">http://www.vbulletin.com/forum/showthread.php?p=1696100#post1696100

Here you can see my nginx + apache/mod_php setup to deal with static + dynamic requests:

http://www.vbulletin.com/forum/showthre … ost1734871">http://www.vbulletin.com/forum/showthread.php?p=1734871#post1734871

With nginx + apache you have both of two worlds and you just proxy to apache php requests. Everything else (static) is managed by nginx.

You should install (if you haven't done yet) an opcode like APC or xCache.

I'm running APC. It helps a lot.

With this setup you can slow down apache to something like this:

StartServers 2
MinSpareServers 2
MaxSpareServers 2
MaxClients 10
MaxRequestsPerChild 1000

This is a very fast and stable setup.

~~http://img189.imageshack.us/img189/8200 … esweek.png">http://img189.imageshack.us/img189/8200/apacheprocessesweek.png" />

~~http://img200.imageshack.us/img200/1848/memoryweek.png" />

Note: This setup works for me and I don't know if will works for you.~~~~

Little Update,

I optimized my php.ini, mysql, and deactivates unnecessary appache modules following these instruction: http://elliottback.com/wp/why-my-wordpr … han-yours/">http://elliottback.com/wp/why-my-wordpress-site-is-so-much-faster-than-yours/

And benchmarked with Apache bench, could not get trough ab -c 100 -n 100

I also installed eAccelerator following these instructions: http://developer.mindtouch.com/Deki/FAQ … 4_(Debian)">http://developer.mindtouch.com/Deki/FAQ/Configuration/HowdoI…InstalleAcceleratoronUbuntu8.04_(Debian)

And now I can get trough ab -c 100 -n 100 easily.

Gonna test more tonight, too much traffic right now. But it seem to have resolve my problem, wanted to put some links for newbies like me out there…

I certainly learned a bunch about configurations in 2 days lol…

I would certainly like to have both apache and ngnix, anyone have a tutorial to get them working simultaniously

Using both at the same time will only waste memory…

Actually, some people like the nginx+apache setup.

They use nginx for all the static content and reverse proxy php pages to apache+mod_php installation.

apache+modphp is more stable than fcgi, you don't necessarily need to modify your existing config and i think modphp offers some features that fcgi doesn't (need to dbl-check that).

Might make sense if your site is heavy on static content and you already have a working configuration that you want to migrate without much hassle.

I am still in the process of installing and configuring things (and learning a lot) on my Slackware 12.2 linode. With nothing but my ssh connection, I am surprise that I still have 300+ mb of free memory on a 360 linode (free -m). I run a Nginx server, three fastcgi php 5.3, mysqld (medium my.cnf), vsftpd, a Git daemon and few other stuff. Everything was compiled from source, I tried to tweak the configure scripts for each program. I have no idea how things will be in a few weeks when everything is in production.

@oliver:

Actually, some people like the nginx+apache setup.

They use nginx for all the static content and reverse proxy php pages to apache+mod_php installation.

apache+modphp is more stable than fcgi, you don't necessarily need to modify your existing config and i think modphp offers some features that fcgi doesn't (need to dbl-check that).

Might make sense if your site is heavy on static content and you already have a working configuration that you want to migrate without much hassle.

Apache + nginx/lighttpd/lightspeed can be more efficient memory-wise when compared to Apache, but nginx/lighttpd/lightspeed alone would always be more memory efficient than both at the same time.

I'm not aware of any stability issues with fastcgi-based dynamic content, and lighttpd is more memory efficient than apache+modphp. With apache+modphp, every single Apache process gets PHP built-in whether it needs it or not. With lighttpd+fastcgi, your main lighttpd process remains PHP-free, handling static content, while you have a configurable number of dedicated PHP processes that you can tailor to suit your load requirements without impacting the memory footprint of static content.

For example, our setup has both PHP and Perl loaded via fastcgi into lighttpd, but our Perl load is way lighter than our PHP load. As such, we've got 6 PHP processes and 2 Perl processes. I believe if you had both PHP and Perl loaded in Apache, you'd have BOTH PHP and Perl in every single Apache process even if they were serving static content!

@Karnius:

I would certainly like to have both apache and ngnix, anyone have a tutorial to get them working simultaniously

In my post above, you can see my setup. You have links with explanations.

@Guspaz:

Using both at the same time will only waste memory…

That's the opposite, because you only need a few apache servers to handle dynamic requests.

All others requests are handles by nginx.

With Apache you have an 25 MB average process to deal with dynamic and static content.

With this setup you have a 3MB process to handle static content.

Since apache only need to handle on request per access you will have a very stable setup.

In my tests, php as fastcgi (with apache or nginx) is slower and you will have problems if you put some load into it.

You can see similar tests here: http://2bits.com/articles/apache-fcgid- … ation.html">http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html

Besides, I like apache configurations and modules, that using this setup I will no loose :)

Moreover, with lighttpd/nginx + PHP fastcgi your FCGI slave processes are always fixed (when you run the spawn-fcgi or whatever to bootstrap them). Whereas with a Apache backend, the number of Apache processes can be dynamic.

That means if you suddenly needs more processes than you have started with, lighttpd/nginx + PHP fastcgi will have to queue, whereas Apache can serve straight away.

Not big deal though if you have planned properly.