Suggestions required - debugging a database connection error
I have a 1GB Linode on which I'm running 4 simple WordPress sites.
I have an intermittent error that occurs about once a week where my MySQL instance stops working and my sites (correctly) return a Database connection error.
I raised this with support and they were very helpful but there's only so much they can do, it's not managed hosting after all.
The results of the MySQL error log file can be seen at the bottom of this post, I can't make much of it.
I thought I'd start by cloning my Linode and moving one site to it and then working as a process of elimination. At least I'd know what WordPress install is causing the problem.
Any thoughts on debugging would be appreciated.
Many thanks.
141030 8:19:48 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
141030 8:19:48 [Note] Plugin 'FEDERATED' is disabled.
141030 8:19:48 InnoDB: The InnoDB memory heap is disabled
141030 8:19:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins
141030 8:19:48 InnoDB: Compressed tables use zlib 1.2.3.4
141030 8:19:48 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
141030 8:19:48 InnoDB: Completed initialization of buffer pool
141030 8:19:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool
141030 8:19:48 [ERROR] Plugin 'InnoDB' init function returned error.
141030 8:19:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141030 8:19:48 [ERROR] Unknown/unsupported storage engine: InnoDB
141030 8:19:48 [ERROR] Aborting
141030 8:19:48 [Note] /usr/sbin/mysqld: Shutdown complete
141030 8:19:49 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
141030 8:19:49 [Note] Plugin 'FEDERATED' is disabled.
141030 8:19:49 InnoDB: The InnoDB memory heap is disabled
141030 8:19:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
141030 8:19:49 InnoDB: Compressed tables use zlib 1.2.3.4
141030 8:19:49 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
141030 8:19:49 InnoDB: Completed initialization of buffer pool
141030 8:19:49 InnoDB: Fatal error: cannot allocate memory for the buffer pool
141030 8:19:49 [ERROR] Plugin 'InnoDB' init function returned error.
141030 8:19:49 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141030 8:19:49 [ERROR] Unknown/unsupported storage engine: InnoDB
141030 8:19:49 [ERROR] Aborting
141030 8:19:49 [Note] /usr/sbin/mysqld: Shutdown complete
141030 8:19:50 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
141030 8:19:50 [Note] Plugin 'FEDERATED' is disabled.
141030 8:19:50 InnoDB: The InnoDB memory heap is disabled
141030 8:19:50 InnoDB: Mutexes and rw_locks use GCC atomic builtins
141030 8:19:50 InnoDB: Compressed tables use zlib 1.2.3.4
141030 8:19:50 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
141030 8:19:50 InnoDB: Completed initialization of buffer pool
141030 8:19:50 InnoDB: Fatal error: cannot allocate memory for the buffer pool
141030 8:19:50 [ERROR] Plugin 'InnoDB' init function returned error.
141030 8:19:50 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141030 8:19:50 [ERROR] Unknown/unsupported storage engine: InnoDB
141030 8:19:50 [ERROR] Aborting
141030 8:19:50 [Note] /usr/sbin/mysqld: Shutdown complete
141030 8:19:50 [Note] /usr/sbin/mysqld: Shutdown complete
141030 8:55:36 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
141030 8:55:36 [Note] Plugin 'FEDERATED' is disabled.
141030 8:55:36 InnoDB: The InnoDB memory heap is disabled
141030 8:55:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
141030 8:55:36 InnoDB: Compressed tables use zlib 1.2.3.4
141030 8:55:36 InnoDB: Initializing buffer pool, size = 128.0M
141030 8:55:36 InnoDB: Completed initialization of buffer pool
141030 8:55:36 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
141030 8:55:36 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
141030 8:55:36 InnoDB: Waiting for the background threads to start
141030 8:55:37 InnoDB: 5.5.40 started; log sequence number 49062462
141030 8:55:37 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
141030 8:55:37 [Note] - '127.0.0.1' resolves to '127.0.0.1';
141030 8:55:37 [Note] Server socket created on IP: '127.0.0.1'.
141030 8:55:37 [Note] Event Scheduler: Loaded 0 events
141030 8:55:37 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.40-0ubuntu0.12.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
2 Replies
@Boomfelled:
(…)
The results of the MySQL error log file can be seen at the bottom of this post, I can't make much of it.
(…)
This is most likely the main issue:
141030 8:19:48 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
141030 8:19:48 InnoDB: Completed initialization of buffer pool
141030 8:19:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool
The InnoDB storage engine can't allocate its main pool (looks like you have 128MB configured) so punts, which then blocks regular the rest of startup since MySQL can't manage any InnoDB tables.
Now, 128MB is not a ton of memory, so it sounds like your Linode may be seriously memory constrained. That might (just a guess) also explain why the database goes away spontaneously, if you get low enough the Linux OOM killer might be killing it off, as it's a long running process that is probably also a heavy memory user.
So my first suggestion would be to look into memory usage. See what "free" shows, check your active processes, and if you have any OOM logs (look for killed process messages, probably in kern.log, or you can look for recent logs on the Lish console).
How are you serving the Wordpress site? If with Apache and modphp, you might have a worker configuration that can use far too much memory under load (default distribution Apache configurations are often horrible for smaller memory configurations, and there are lots of posts in the forums about tuning Apache configurations for such a case), or if php-fpm you might have a pool definition with a maximum servers too high. Essentially, check roughly how much memory each Wordpress process (Apache worker with modphp, or php-fpm pool process) is using, then figure out how many requests you should let run in parallel and make sure your worker limit or php-fpm pool limit stays below that.
– David
That's a lot to take in and will take a while to investigate, but I'll follow your suggestions and work through them.
Thanks again for taking the time to help me out.