Linode unresponsive, brute force attack?
Hi guys,
Recently my linode got unresponsive, I couldn't ssh/ping the linode at all. Checking the lish console I could saw that I was being flooded with constant firewall messages, being unresponsive to anything I tried to type, I mean, I couldn't type anything at all… it was just printing the messages bellow (these are the full length messages, from the dmesg command - since lish console doesn't show the full lenght):
[17709.700751] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=106.51.2.182 DST=<my-linode-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=109 ID=13249 DF PROTO=TCP SPT=60794 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
[17808.396955] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=188.166.242.119 DST=<my-linode-ip> LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=14988 PROTO=TCP SPT=61953 DPT=2011 WINDOW=1024 RES=0x00 SYN URGP=0
[17818.344859] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=113.176.95.237 DST=<my-linode-ip> LEN=52 TOS=0x08 PREC=0x40 TTL=107 ID=30809 DF PROTO=TCP SPT=55920 DPT=1433 WINDOW=8192 RES=0x00 SYN URGP=0
[17654.481595] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=74.120.14.90 DST=<my-linode-ip> LEN=44 TOS=0x00 PREC=0x20 T
TL=42 ID=1242 PROTO=TCP SPT=13278 DPT=54321 WINDOW=1024 RES=0x00 SYN URGP=0
[17692.504090] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=123.24.157.11 DST=<my-linode-ip> LEN=52 TOS=0x08 PREC=0x40 TTL=106 ID=17473 DF PROTO=TCP SPT=65100 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
[17694.943237] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=220.247.242.7 DST=<my-linode-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=112 ID=24822 DF PROTO=TCP SPT=54526 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
[17696.225263] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=200.54.150.42 DST=<my-linode-ip> LEN=52 TOS=0x08 PREC=0x40 TTL=112 ID=40356 DF PROTO=TCP SPT=54034 DPT=445 WINDOW=65520 RES=0x00 SYN URGP=0
[17709.700751] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=106.51.2.182 DST=<my-linode-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=109 ID=13249 DF PROTO=TCP SPT=60794 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
[17720.983329] iptables denied: IN=eth0 OUT= MAC=f2:3c:92:a5:64:e0:62:3d:e6:d9:49:b0:08:00 SRC=187.147.68.27 DST=<my-linode-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=116 ID=20333 DF PROTO=TCP SPT=37624 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0</my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip></my-linode-ip>
The only option I was left was to reboot the linode, which "solved" the problem.
Can anybody help me decipher what was going on here? Was I under some kind of attack? This was preventing all web traffic to my linode.
This is a CentOS 8 instances and I have fail2ban + nftables configured with all ports closed except for ssh, http and https.
The nftables rules are as follow:
table ip my_nft {
chain INPUT {
type filter hook input priority 0;
# accept any localhost - loopback (lo0) - traffic and drop all traffic to 127/8 that doesn't use lo0
iifname "lo" counter accept
ip daddr 127.0.0.0/8 counter reject
# Accept traffic originated from us
ct state established,related counter accept
ct state invalid drop
# Allow SSH, HTTP and HTTPS connections
tcp dport {ssh,http,https} counter accept
# Allow ping
icmp type echo-request counter accept
# Log iptables denied calls
limit rate 5/minute burst 5 packets counter log prefix "iptables denied: " level debug
# Drop all other inbound - default deny unless explicitly allowed policy
counter drop
}
chain FORWARD {
type filter hook forward priority 0;
# Drop all other inbound - default deny unless explicitly allowed policy
counter drop
}
chain OUTPUT {
type filter hook output priority 0;
# Drop all other inbound - default deny unless explicitly allowed policy
counter accept
}
}
Any help would be very appreciated, thanks!
4 Replies
I don't think this is anything special. I get thousands of these an hour. Your example comes from IP addresses in Vietnam, India, the US, Mexico. Pretty unremarkable actually…
Since you say that re-booting "solved" your problem, I would start looking elsewhere for an actual compromise or a stuck service. If you find one, you should correct it. You can always log in at the console with Glish.
For the messages listing DPT=445, the attacker is trying to get access to the Micro$oft CIFS service (aka Samba):
microsoft-ds 445/tcp # Microsoft Naked CIFS
microsoft-ds 445/udp
The others are just random port scanning.
One thing you didn't give in your example is any of the timestamps so I have no clue how frequent this stuff is. That would have been helpful. These console messages are stored in /var/log/kern.log.
The only advice I can give is to DROP traffic you don't accept and not REJECT it. REJECT provides the sender an indication that you exist after the traffic is REJECTed. It looks like you do that already. Beware that DROPping traffic may cause timeouts on your rate-limited services so there's no real blanket rule about that.
I see the following:
- You may want to rate-limit ssh, http, https & ping.
I don't know if you care about IPv6 but you should…
- You should add ::1 to the list of IP addresses you only allow from lo0.
- If you want to accept ICMP echo requests, you should accept ICMP6 version as well. You can test this with the ping6 command.
- If you want to play nicely with Linode's IPv6 infrastructure, you need allow IPv6 Neighbor Discovery and IPv6 Router Discovery. These are both ICMP6 messages too. You can segregate these in a separate chain if you like.
You can Google solutions to all these for your setup (mine is different, otherwise I would have posted it for you).
Life with public IP addresses…
-- sw
Thank you very much for your feedback @stevewi !
I fully agree with your comments and I tend to believe I had a problem somewhere else, indeed. The reason for my "alarm" was the fact that I wasn't expecting to saw these kind of logs on the Glish console itself.
I'm sorry but I don't have the original full logs from dmesg anymore (I don't have the kern.log activated). The way I got these were from the dmesg command itself. Although, judging for today's one, I would say I got those about 5-6 times/minute.
Currently I'm not looking into IPV6 yet, although is something that I have planned to do in a near future!
Thank you once again for your feedback! I do really appreciate!
Thanks!
@JayS --
You write:
Currently I'm not looking into IPV6 yet, although is something that I have planned to do in a near future!
The actions I've outlined just make your Linode play nice in Linode's IPv6 infrastructure. You should probably implement them whether or not you plan to actually use IPv6.
-- sw