Disk I/O Issue - Site Crashing
LINODE 1536
Running Drupal website :
LAMP + Memcache
I am a LINUX NOOB - My site is clearly misconfigured some where -
It runs fine and then all of the sudden it starts swapping until you can't even terminal in. I have seen maybe 10 users on the site at a time. Here is my top:
top - 08:00:09 up 2:07, 1 user, load average: 0.14, 0.22, 0.40
Tasks: 76 total, 1 running, 75 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.7%us, 2.8%sy, 0.0%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.1%st
Mem: 1541716k total, 1045544k used, 496172k free, 7992k buffers
Swap: 262140k total, 37348k used, 224792k free, 816504k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1943 mysql 20 0 150m 46m 2232 S 43 3.1 39:32.65 mysqld
3484 www-data 20 0 54040 26m 4828 S 12 1.8 0:03.66 apache2
3477 www-data 20 0 53984 25m 4284 S 9 1.7 0:03.12 apache2
3469 www-data 20 0 52936 24m 4312 S 8 1.7 0:08.17 apache2
1963 nobody 20 0 71612 9196 360 S 0 0.6 0:05.85 memcached
3478 www-data 20 0 52696 24m 4300 S 0 1.6 0:03.34 apache2
3485 dave 20 0 2464 1056 824 R 0 0.1 0:00.07 top
1 root 20 0 2720 520 316 S 0 0.0 0:00.60 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/0
4 root 20 0 0 0 0 S 0 0.0 0:03.57 kworker/0:0
5 root 20 0 0 0 0 S 0 0.0 0:00.06 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
8 root 20 0 0 0 0 S 0 0.0 0:00.30 kworker/1:0
9 root 20 0 0 0 0 S 0 0.0 0:00.15 ksoftirqd/1
10 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
free -m
total used free shared buffers cached
Mem: 1505 1275 229 0 10 693
-/+ buffers/cache: 571 934
Swap: 255 34 221
KeepAlive set to OFF
MinSpareServers 3
MaxSpareServers 6
MaxClients 12
MinSpareThreads 3
MaxSpareThreads 6
Thread Limit 20
MaxClients 12
MaxSpareThread 20
MaxClientsChild 20
4 Replies
* Fine Tuning
#
key_buffer = 16K
sortbuffersize = 4M
joinbuffersize = 4M
readrndbuffer_size = 512K
maxallowedpacket = 1M
thread_stack = 64K
threadcachesize = 256
This replaces the startup script and checks MyISAM tables if needed
the first time they are touched
myisam-recover = BACKUP
max_connections = 20
tmptablesize = 32M
maxheaptable_size = 32M
table_cache = 256
thread_concurrency = 8
THANKS- Iappreciate the help!
I have a drupal site. prefork was very fast for me, but the swap was crashing the site with like 10 users.
So I did some poking around and was able to switch from fork to worker. Made a monster difference.
Here is my graph breakdown. I am using higher CPU. Even got an alert about it.
You'll see that the i/o is still not zero, but pretty much. (i think on linux there is always a little see linuxatemyram.com )
Anyway - switching the way i did required CGI- made a quick apt get for that and made a few changes to virtual host file per a tutorial I found for another version of Linux but it was close enough to get working.
That fixed my server but my site was downloading the page when ever you clicked a link so I had to set the base URL on my Drupal 6 website (simple uncommented a line in the settings.php file and altered the example.com)
I will post new worker set up asap to help anyone else out.
If you want my terminal notes from my cgi addition hit me up- @daveferrara1
and wholla!
Another cheap win is an object caching module like APC, the Alternative PHP Cache. When set up correctly it has zero side-effects but speeds up execution (and reduces CPU load).
sudo apt-get install libapache2-mod-fcgid