Why is my idle linode generating traffic and is using CPU?
Hi,
I set up a linode, two days ago. Its a CentOS machine so firewalld is eanbled by default.
The first thing when I set up the vm, I created a user and added this user to the wheels group. Then I added my public ssh key for that user. Afterwards I disabled root login and password authentication in my ssh config. I also only allow that particular user to login. I tested and It works, the only way in is with my pub key.
Now, I did not ssh into the machine for almost a day. When I check my dashboard I see there was a traffic spike and the CPU was also used.
<img alt="cpu usage and traffic" src="https://user-images.githubusercontent.com/39703898/81266163-2fa4fa00-903c-11ea-9329-add406e4f03d.png">
Am sure that the spike at 20:00 was not me. Am even more certain that the spike from 12:00 to approx 4:00 wasn't me. I was sleeping at that time.
Does anyone know why how this is possible?
3 Replies
@bluebrown --
You write:
Am sure that the spike at 20:00 was not me. Am even more certain that the spike from 12:00 to approx 4:00 wasn't me. I was sleeping at that time.
It probably wasn't you. If you look at the graph, you have a spike of public inbound traffic during the hours 0000 to 0400 as well. That was probably the trigger. These hours are peak hours for port scanners and hackers looking for vulnerabilities…particularly from the Russians, the Chinese & the Koreans (both north and south) and all their proxies around the world.
What I would do is:
First, look at your firewall configuration and shut down any port for any service you don't actually use. Just having firewalld running is not any kind of guarantee that it's stopping anything…you have to tell it what to stop. Remember that you are always using DNS!
Second, severely restrict ICMP (ping[6] among other things). That will largely eliminate ping sweeping. You have to be careful about this because ICMP has a legitimate role in router discovery.
Third, implement fail2ban(1) to start looking for abusers in the logs for the services using ports left open (ssh(1), mail, www, etc). There is some lag with this but it's effective.
Lastly, you can implement blacklisting. Form lists of offender IP addresses/networks and install these lists into your firewall. Institute a firewall rule that DROPS ALL traffic from any member of the list (typically, you have to segment these lists into nodes and networks for storage efficiency…so really you'll have 2 lists…4 if you include IPv6). FWIW, here are the sizes of my current blacklists in lines:
46084 bl-nets4.conf
19507 bl-nets6.conf
157600 bl-nodes4.conf
7 bl-nodes6.conf
Each line represents a node or network. Networks are expressed in CIDR notation so the number of actual nodes blocked is many times greater that what's shown here. I block ALL traffic from 22 countries (and several foreign subsidiaries of some -- e.g. US operations of ChinaNet Mobile).
Don't forget IPv6! You don't give an IPv6 traffic graph but I would imagine that it looks similar to the IPv4 one.
I hope this helps. Respond back if you need help with any of this…I've implemented them all. You may have to give up firewalld and learn the iptables/nftables backend to do some of this stuff…or figure out how to augment firewalld's rules with your own.
-- sw
FWIW, @stevewi has covered this off nicely.
I just wanted to add that (although I don’t think it’s the case here) there may also be system-generated automated tasks that run at various times.
Ubuntu, for example, has default scheduled tasks to run maintenance, check for package updates, news etc. This can also contribute to CPU usage and/or bandwidth.
These would just be short-lived spikes rather than continuous usage like you’re seeing at 12-4am.
