Please help: Piwik, Mysql crash + High Disk IO Alert
Details:
Running Debian 7 Latest 64 bit on a 24320 MB Linode
Running apache2 to serve up a php 5.4.4 web app -
Installed Piwik via the Linode documentation - installed mysql in the course of doing so - everything ran smoothly for about 3 months - never any problems or slowness.
Added a couple of long running node scripts on the same box.
Got a new relic alert that memory usage was over 95% - assumed this was wrong, linode manager shows an average of 4% cpu - nothing on the box is really running at a scale yet that should cause any problems.
Then today I go to hit my piwik analytics page and I get an error: cannot connect to mysql - so mysql crashed - was likely responsible for the thrashing now visible on linode manager io panel.
Go to try and start mysql manually= [FAIL] init.d/mysql: ERROR: The partition with /var/lib/mysql is too full! … failed!
Filesystem Size Used Avail Use% Mounted on
rootfs 24G 6.3G 16G 29% /
/dev/root 24G 6.3G 16G 29% /
devtmpfs 493M 0 493M 0% /dev
tmpfs 99M 184K 99M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 249M 0 249M 0% /run/shm
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 1319360 83692 1235668 7% /
/dev/root 1319360 83692 1235668 7% /
devtmpfs 126139 1442 124697 2% /dev
tmpfs 126406 793 125613 1% /run
tmpfs 126406 3 126403 1% /run/lock
tmpfs 126406 2 126404 1% /run/shm
Meanwhile there is nothing in my var/log/mysql - all the mysql related log files are empty
I really liked Piwik, but from browsing around it seems other Linode users have had issues with it - and there's talk of a memory leak.
I would love to have analytics that are not harvesting all my data for the empire (google) but even more importantly I'd like to know what happened here so I can prevent it in the future.
Any thoughts or suggestions?
9 Replies
nobody?
If you think it's piwik have you looked at your table sizes & record counts? Are they high? Have you archived some of your activity (I forget what they call the processes; our designer takes care of the piwik admin)?
After running Piwik for a while I added a wordpress install - which is also hosed now due to not being able to start mysql
Not sure what to do about being unable to start mysql and getting a space error despite having plenty of disk space…
MySQL can (and will) die if it runs out of RAM. What does your MySQL log say (from around the time of the problems)? How big of a Linode are you running? What do you get when you run "free -m"?
total used free shared buffers cached
Mem: 987 971 16 0 1 26
-/+ buffers/cache: 943 44
Swap: 255 255 0
Still havent claimed my full sized linode yet.
All of my mysql logs are empty, including the one explicitly set in my.cnf as well as syslog
this thread
I forgot about this potential RAM issue with Wordpress. If you're running Wordpress it could be the 'Max RAM' setting isn't defined so it tries to take 256M. Look at this thread