Disk filling up, cannot see why!
Hi,
I have been running a linode server, setup via (Laravel) Forge service for a few years. No issues.
Last week I performed an upgrade to the CMS that's running on it. All went (seemingly) smoothly, but within a day the server was throwing an error and my site wouldn't load. Managed to find an error that pointed to no disk space left.
I couldn't work out where the space was being used up so deleted the server and provisioned a fresh new one.
I re-added the single site to this server. (It's approx 6Gb in size as it has a fair few photo assets.) All seemed well. I checked the next morning on the disk and it was showing that the main disk was using 12Gb. Within a day (this morning) it was showing 16Gb usage. It's now (7 hours later) showing 18Gb usage.
The main site folder hasn't changed size at all and is still roughly 6Gb in size.
I'm struggling to see where this space is getting eaten up by.
I've checked folder sizes from the root and get the following…
swapfile 1.1Gb
usr 1.7Gb
var 1.0Gb
lib 1.1Gb
home 6.1Gb
The rest of the folders give me permissions errors.
Any idea on where best to check? At this rate the new 80Gb disk will be full in a week or two!
14 Replies
@bbigger - thanks for that, that's really helpful.
So, running that it appears that it's the /var/log/mysql folder that is the culprit. It's showing as 6.9G in size.
Not sure where I go from here though?
bbigger
Linode Staff
@shorn Happy to help. That folder holds your MySQL binary log files, which are important for recovery and high availability applications. Accordingly, you could disable logging or purge logs if you're using them for replication. This post from nixCraft does a great job of explaining MySQL binary log files and how to handle them.
If your binary logs are important to maintain, then you should consider resizing your Linode to increase disk space for allocation or otherwise freeing disk space from other areas.
@bbigger - so managed so view the folder as root. That's right, seems to be a lot of binary log files. 70 to be exact that are all 101M in size.
I'm not quite sure why these are being produced in this volume though. Could this be something to do with the CMS (CraftCMS) running on it?
Seems strange as I set up a handful of sites with a similar setup and not hit this before
I'm wondering if this volume of logs could indicate an error with my database?
bbigger
Linode Staff
You can display the contents of MySQL binary logs with mysqlbinlog. Here's more info on the utility from MySQL's documentation. Given that the output will likely be very large, it would probably help to write the output to a file like so:
mysqlbinlog example-bin.000001 > example.log
I'm not familiar with CraftCMS, but it looks like they have several community resources that may be helpful.
@bbigger - thanks for that. I've been able to create the file. Any idea on how I can download that file locally? (Sorry, no used to this level!).
So, since posting this, I now have 76 of these files. It's looking like I'm going to need to disable the logging, but I'm wary of doing this without understanding why they are being created.
Not sure how I can go about working out why they are being created in this kind of volume!?
bbigger
Linode Staff
@shorn sure thing! You could copy files locally using scp (see this man page for details) or rsync (here's our guide on that) among several other options.
Examining the saved outputs from the binary log files should show you the database changes, which could lead you to the cause of the issue. In the meantime, you could keep future log files under control by setting them to automatically expire.
@bbigger- that's great, I've managed to get it via rsync.
To be honest, I can't see why any obvious reason in the log. The odd thing is that I've created quite a few sites using Craft and never hit this issue before so it's a strange one.
What's responsible for writing these logs? i.e what action/event will cause it to be logged? Obviously it's related to the database. Is it changes that are getting written?
OK, so I think I might have worked out what's happening. This started since I upgraded the CMS. This brought about a large number of deprecation errors (~400).
Although these are initially stored in the database, the 'last occurrence' gets logged in the db each time it's hit. The logs show multiple INSERT INTO craft_deprecationerrors entries. This means that these are getting updated nearly every second as the site always has traffic and every visitor is likely to hit a deprecation error.
Sound like a feasible explanation?
I am not an expert, yet. More of an enthusiast… But that makes sense in my head. Especially if the log is saying what is it saying…
What's responsible for writing these logs? i.e what action/event will cause it to be logged? Obviously it's related to the database. Is it changes that are getting written?
Binary logs contain any changes being written to the database, as they happen.
https://dev.mysql.com/doc/refman/8.0/en/binary-log.html
And so every time there is an update or an insert - it gets recorded in a binary log.
Typical use cases are 1) replication (between multiple databases) 2) backup (one of the ways to do it).
Binary logs can be turned off (and that's the default, so you must have a config setting… wonder why).
If you wish to be cautious and keep them enabled - you may want to trim them while you work out the underlying issue:
https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html
Just be sure to not delete the files directly.
I'd also look into disabling those "INSERT INTO craft_deprecationerrors" in your CMS code if possible - I mean at this point you already know that some of your site code needs to be updated, and logging these over and over again is only causing trouble.