Nanode running Mastodon instance on Docker becomes unresponsive
I was digging into this topic (Linode running Docker keeps becoming unresponsive) because I've been having similar issues.
I'm running a Mastodon instance from the marketplace on (initially) a 2GB Nanode. It's gone down in this way twice over the last couple of weeks.
Tail-ing the daemon.log file didn't give me anything useful, but looking at the end of daemon.log.1 —the logs before it was rotated— shed some insight:
Aug 20 00:00:01 social mandb[64586]: /usr/bin/mandb: can't write to /var/cache/man/64586: No space left on device
Aug 20 00:00:01 social mandb[64586]: /usr/bin/mandb: can't create index cache /var/cache/man/64586: No space left on device
Aug 20 00:00:01 social systemd[1]: man-db.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 20 00:00:01 social systemd[1]: man-db.service: Failed with result 'exit-code'.
Aug 20 00:00:01 social systemd[1]: Failed to start Daily man-db regeneration.
Quite simply, the instance is running out of disk space. Bumping this temporarily to a 4GB Nanode has gotten everything working again.
I'd like to investigate the following:
- What's taking up all this space (this is a low-volume, two-user Mastodon instance, but my guess is media)
- How to mitigate that so I don't have to keep bumping to a higher-tier Nanode
Any tips you all might be able to provide would be much appreciated!
3 Replies
jhartman
Linode Staff
As you have already discovered based on that daemon's error logging, this appears to be a related to Mastodon's heavy writing of media and temporary files. I have found some discussion about this issue on Reddit:
- My (single user) instance uses >20GB disk space? : r/Mastodon
- Ubuntu 18.04 - Full disk crushed my mastodon installation, help! : r/Mastodon
Which contain recommendations for Best Practice file-storage tips and a blog-post that provides some commands and Cron Jobs that can be setup to automatically wipe these files:
By setting up this recurring task to delete these excess files, you should be able to prevent this from being an ongoing issue. That said, although you haven't seemed to run across any memory issues so far, you do have the potential to encounter those problems if your user base increases.
Mastodon's own Github repo suggests that Mastodon running within a Docker container needs around 2.2GB of RAM to run. Aside from upgrading to a plan with more memory, I found a different blog with information on how to optimize Mastodon to use less memory.
tlambert
Linode Staff
You can also check out this guide for instructions on setting up Object Storage with your Mastodon instance. After creating a bucket, you'll need to reconfigured the .conf file found in your /etc/nginx/sites-available directory as well as the /home/mastodon/live/.env.production file. There are also instructions for syncing your existing data.
