Mariadb server won't restart
I saw that my website was down, and so I foolishly restarted my Linode. (Debian 9, running MariaDB and a PHP website). Now I see that the problem was that my database was non-responsive. I can't get my MariaDB back up again. When I run systemctl restart mariadb
, I get: "Job for mariadb.service has failed because the control process exited with error code. See systemctl status mariadb.service and journalctl -xe for details."
Output of systemctl status mariadb
:
● mariadb.service - MariaDB 10.1.44 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-08-29 19:46:57 CDT; 2min 23s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 4810 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 4718 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery
; [ $? -eq 0 ] &&
Process: 4712 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 4711 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 4810 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"Aug 29 19:46:53 mnminimarket systemd[1]: Starting MariaDB 10.1.44 database server…
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Warning] option 'table_open_cache': unsigned value 335544
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1)
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Warning] Could not increase number of max_open_files to m
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 29 19:46:57 mnminimarket systemd[1]: Failed to start MariaDB 10.1.44 database server.
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Unit entered failed state.
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Failed with result 'exit-code'.
Output of journaltcl -xe
:
Aug 29 19:46:53 mnminimarket systemd[1]: Starting MariaDB 10.1.44 database server…
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd-- Support: https://www.debian.org/support
-- Unit mariadb.service has begun starting up.
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Warning] option 'table_open_cache': unsigned value 335544
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1)
Aug 29 19:46:54 mnminimarket mysqld[4810]: 2020-08-29 19:46:54 139732841295232 [Warning] Could not increase number of max_open_files to m
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 29 19:46:57 mnminimarket systemd[1]: Failed to start MariaDB 10.1.44 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd-- Support: https://www.debian.org/support
-- Unit mariadb.service has failed.
-- The result is failed.
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Unit entered failed state.
Aug 29 19:46:57 mnminimarket systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 29 19:49:20 mnminimarket sudo[4839]: amanda : TTY=pts/0 ; PWD=/home/amanda ; USER=root ; COMMAND=/bin/systemctl status mariadb
Aug 29 19:49:20 mnminimarket sudo[4839]: pam_unix(sudo:session): session opened for user root by amanda(uid=0)
Aug 29 19:49:35 mnminimarket sshd[4842]: Received disconnect from 222.186.31.83 port 64565:11: [preauth]
Aug 29 19:49:35 mnminimarket sshd[4842]: Disconnected from 222.186.31.83 port 64565 [preauth]
Aug 29 19:49:47 mnminimarket sshd[4844]: Connection closed by 134.175.249.84 port 49370 [preauth]
Aug 29 19:50:00 mnminimarket sudo[4839]: pam_unix(sudo:session): session closed for user root
I've tried several of the fixes on this page, https://mariadb.com/kb/en/what-to-do-if-mariadb-doesnt-start/, namely ProtectSystem=Off
, TimeoutStartSec=infinity
, TimeoutStopSec=infinity
.
I have a hunch it has something to do with the fact that I was getting a 'Too many connections' error in my logs sometimes before the timeout. But now I can't even get my db to start at all to try to fix that issue.
1 Reply
Could not increase number of max_open_files
I suspect this may be the reason. Have you previously tried to increase the max number of connections or max number of open tables, as this would require MariaDB to request more open files - and it may be trying to request more than the OS will allow.
See this warning on MariaDB’s website:
Make sure that your operating system can cope with the number of open file descriptors required by the table_open_cache setting. If table_open_cache is set too high, MariaDB may start to refuse connections as the operating system runs out of file descriptors. Also note that the MyISAM (and Aria?) storage engines need two file descriptors per open table.