High disk io and mysql not working
I'm running linode 1gb, ubuntu+apache+mysql+php5, all installed and optimized per linode instructions.
Everything was working great till today, once I started to get these emails "notification threshold (1000) for disk io rate by averaging 5351.16".
Today I transferred my main, biggest site here, and 2 hours later, troubles started.
All sites are wordpress, and it can't connect to db (all sites), then mysql crashes, everything becomes slow (even terminal, it takes 5min to login via ssh and switch to root).
When I run service mysql status, I get this: post-start process 16000. I tried optimizing db's with: myisamchk –recover /var/lib/mysql//.MYI, didn't help.
What could be the issue? I can't even start iotop now, takes forever…
Here's the screenshot once it's all slow:
~~![](<URL url=)http://s7.postimg.org/8vkuj848r/iotop.jpg
and here's when it works (guess mysql is dead here):
~~![](<URL url=)http://s8.postimg.org/thx0bblpx/iotop2.jpg
Can someone help, I can't catch a breath today…~~~~
13 Replies
The Swap column looks bad while you're having the problems. When you log in via SSH or LISH do you get Out of Memory (OOM) errors?
![](
While this is happening run "top". While top is running press shift+f (for a capital F) and you'll get a sort menu. Chose 'n' to sort by memory. When you're done press 'q' to quit.
[Sun Oct 27 06:31:19 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 04:40:29 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 04:40:29 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:01:22 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 06:01:22 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:01:56 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 06:01:58 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:27:05 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 06:27:06 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 08:36:46 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 08:36:46 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 08:45:46 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 28 09:56:08 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 10:01:32 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 09:58:16 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 28 12:01:01 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 11:57:44 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
Mysql:
131028 14:32:44 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecate$
131028 14:32:44 [Note] Plugin 'FEDERATED' is disabled.
131028 14:32:44 InnoDB: The InnoDB memory heap is disabled
131028 14:32:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131028 14:32:44 InnoDB: Compressed tables use zlib 1.2.3.4
131028 14:32:44 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
131028 14:32:44 InnoDB: Completed initialization of buffer pool
131028 14:32:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
131028 14:32:44 [ERROR] Plugin 'InnoDB' init function returned error.
131028 14:32:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131028 14:32:44 [ERROR] Unknown/unsupported storage engine: InnoDB
131028 14:32:44 [ERROR] Aborting
131028 14:32:44 [Note] /usr/sbin/mysqld: Shutdown complete
Any other log I should paste?
~~![](<URL url=)http://s21.postimg.org/ylyu6s97b/mem2.png
I don't think I need bigger linode, there's around 2k visits all together daily.~~
You may have too many of something (Apache workers, MySQL InnoDB memory pool, etc). There are several recent threads regarding Apache related resource usage. Perhaps the Wordpress forums have threads on common optimizations to run Wordpress on a 1GB VPS.
@pex23:
I don't think I need bigger linode, there's around 2k visits all together daily.
&
Maybe you're right, but once site was up - everything was running smoothly. I don't believe I got massive traffic all of the sudden.
You may not need a larger Linode in the long run with only 2k v/p/d. But until you figure out how to get this to run properly in less than 1GB of RAM upsizing may be a good solution.
I counted at least 22 apache processes and at least 17 kworker processes. You're also requesting 128.0M for the InnoDB buffer pool. Reducing the Apache & kworker processes, as well as the size of the InnoDB pool (unless you have a lot of DB content), will definately work to your advantage. If your Wordpress DB isn't using the InnoDB driver - which is a possibility - than you certainly don't want to allocate RAM for InnoDB.
I forgot about this potential RAM issue with Wordpress. If you're running Wordpress it could be the 'Max RAM' setting isn't defined so it tries to take 256M. Look at this thread