Suspicious activity: was it a hack, and am I doing the right things to prevent it happening again?
Hi,
At 2am local time my server had a 200% CPU usage spike, and from 2am onwards it was a constant plateau of 100% usage. All the sites on my server were down (Ubuntu 20.04, all Wordpress sites).
I immediately rebooted the server but this did nothing. Then I panicked. After panicking I set the server to rebuild from a previous backup, and deployed a backup to a new Linode in case the first was compromised in a way that rebuilding from a backup would not resolve.
After rebuilding and turning my original server back on, all my sites are up again. I have since -
- Disabled login via root (although I can still login via root on Putty so I'm unsure whether I need to do something separate to prevent root access there?)
- Changed root password to a much stronger password
- Set up a non-root sudo level login with a very strong password
- Installed Fail2ban (although I'm unsure to test whether it's working?)
I would like to understand whether there are any other things I should be doing right away to secure my Linode, and how to tell whether -
- It was a hack that caused the issue
- If so whether the server is still compromised
Any help is much appreciated, thank you. If I can provide any extra info to help answer my question please let me know.
Chris
1 Reply
hdiep
Linode Staff
Based on what you've described, this could be a system compromise since you were able to rebuild your server and your website has stabilized since then. Although we can't definitely say for sure because a process could have gotten stuck and caused your CPU to spike. Since you rebuilt your Linode, you won't have any logs that can help you investigate whether there was an issue with your internal processes or a serial brute force attempt on your Linode. In case it is a compromise, the steps you've taken so far to secure it are a great first step. If you're trying to investigate whether your server was recently compromised and is still compromised, we have a few guides that can assist with this:
- I've noticed some suspicious activity on my Linode, what do I do?
- Recovering from a System Compromise | Linode Docs
We also have other guides that can provide you step by step instructions on how to harden your server:
- How to Set Up and Secure a Compute Instance | Linode Docs
- What is an Easy Way to Secure My Server? (19349) | Linode Questions
In case you may be interested on how you can secure your Linode account as well, I provided a detailed guide below that provides some best practices on how to prevent unauthorized account access:
Lastly, I highly recommend investigating your CPU usage as there could be many reasons why your Linode is would suddenly using more CPU. If this happens again, you could try logging onto your Linode during one of these periods of increased CPU usage and confirming your usage with a utility such as htop. If you're using Apache, you may want to even fine tune your Apache server.I suggest reviewing these resources below on how you can investigate the cause of high CPU usage: