constant IO
2 days ago I started getting a sustained 1kb/s in, and 1 out. Now it's up to 1kb/s in and 2 out.
I only have a few scattered search engine spiders in my nginx log files, certainly not enough for sustained traffic 24/7. I noticed more ssh hacking attempts, but I've added a few ip addresses to my firewall and most are getting blocked.
How can I figure out this spike in outbound network traffic?
So far I've viewed netstat -a output (which didn't have anything malicious), and I've checked on the logs to see what's getting updated:
ls -alt /var/log
Am I being paranoid? Want to make sure my box isn't crapping out spam emails or something worse.
14 Replies
Now I need to learn how to understand all of it's output, I'm off to tutorial land
update-rc.d -f ntop remove
@obs:
iotop is for disk usage, they want network usage hence ntop
Oops, didn't catch that, I saw "i/o" and thought disk "i/o", not network activity. I retract my former comment.
@Azathoth:
Aggressive port scanners? If inbound traffic roughly equals outbound my first guess is firewall reject packets. Try changing default policy to drop (no response to invalid inbound) and see if something changes.
That sounds like a fantastic idea! I'm using ufw on ubuntu 11.04 as a front end to iptables. The default is deny from all, and here is the ruleset:
# ufw status numbered
Status: active
To Action From
-- ------ ----
[ 1] Anywhere DENY IN 91.207.234.129
[ 2] Anywhere DENY IN 216.176.62.137
[ 3] Anywhere DENY IN 222.143.26.249
[ 4] 22 ALLOW IN Anywhere
[ 5] 80 ALLOW IN Anywhere
[ 6] Apache Full ALLOW IN Anywhere
[ 7] 3000 ALLOW IN <redacted></redacted>
My understanding was these rules and the ufw default of deny all should prevent port scanning network traffice, is there a way for me to check? I'll research how to properly block port scanning, as always any pointers are greatly appreciated. (the open port 3000 is for ntop, recently installed to help me find out my network IO culprit)
iptables -L -n -v
It'll output a load of text (which you can put here).
Look for lines with DROP or REJECT in them, especially one starting "Chain INPUT"
If it says "DROP" then it's being dropped and no response is sent, if it says "REJECT" then it means it's sending notification back.
Entire output:
# iptables -L -n -v
Chain INPUT (policy DROP 65 packets, 4400 bytes)
pkts bytes target prot opt in out source destination
991K 148M ufw-before-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
991K 148M ufw-before-input all -- * * 0.0.0.0/0 0.0.0.0/0
13393 690K ufw-after-input all -- * * 0.0.0.0/0 0.0.0.0/0
12014 621K ufw-after-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
12014 621K ufw-reject-input all -- * * 0.0.0.0/0 0.0.0.0/0
12014 621K ufw-track-input all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ufw-before-logging-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-before-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-after-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-after-logging-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-reject-forward all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
912K 189M ufw-before-logging-output all -- * * 0.0.0.0/0 0.0.0.0/0
912K 189M ufw-before-output all -- * * 0.0.0.0/0 0.0.0.0/0
170K 13M ufw-after-output all -- * * 0.0.0.0/0 0.0.0.0/0
170K 13M ufw-after-logging-output all -- * * 0.0.0.0/0 0.0.0.0/0
170K 13M ufw-reject-output all -- * * 0.0.0.0/0 0.0.0.0/0
170K 13M ufw-track-output all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-after-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-input (1 references)
pkts bytes target prot opt in out source destination
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:137
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:138
2 96 ufw-skip-to-policy-input tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:139
10 496 ufw-skip-to-policy-input tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:445
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ufw-skip-to-policy-input all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST
Chain ufw-after-logging-forward (1 references)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '
Chain ufw-after-logging-input (1 references)
pkts bytes target prot opt in out source destination
71 4648 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '
Chain ufw-after-logging-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-forward (1 references)
pkts bytes target prot opt in out source destination
0 0 ufw-user-forward all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-before-input (1 references)
pkts bytes target prot opt in out source destination
46 3885 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
140K 23M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
15 656 ufw-logging-deny all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
15 656 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 3
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 4
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 11
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 12
5 172 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 8
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
19415 1163K ufw-not-local all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT udp -- * * 0.0.0.0/0 224.0.0.251 udp dpt:5353
19415 1163K ufw-user-input all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-before-logging-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-logging-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-logging-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-output (1 references)
pkts bytes target prot opt in out source destination
46 3885 ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
137K 28M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
11930 867K ufw-user-output all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-logging-allow (0 references)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW ALLOW] '
Chain ufw-logging-deny (2 references)
pkts bytes target prot opt in out source destination
12 520 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID limit: avg 3/min burst 10
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '
Chain ufw-not-local (1 references)
pkts bytes target prot opt in out source destination
19415 1163K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type MULTICAST
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST
0 0 ufw-logging-deny all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-reject-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-reject-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-reject-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-skip-to-policy-forward (0 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-skip-to-policy-input (7 references)
pkts bytes target prot opt in out source destination
12 592 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-skip-to-policy-output (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-track-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-track-output (1 references)
pkts bytes target prot opt in out source destination
12 720 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
11917 866K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
Chain ufw-user-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-user-input (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 91.207.234.129 0.0.0.0/0
0 0 DROP all -- * * 216.176.62.137 0.0.0.0/0
5619 337K DROP all -- * * 222.143.26.249 0.0.0.0/0
6110 367K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:22
37 2092 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:80
1 48 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 /* 'dapp_Apache%20Full' */
187 9724 ACCEPT tcp -- * * <redacted> 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT udp -- * * <redacted> 0.0.0.0/0 udp dpt:3000
Chain ufw-user-limit (0 references)
pkts bytes target prot opt in out source destination
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 5 LOG flags 0 level 4 prefix `[UFW LIMIT BLOCK] '
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain ufw-user-limit-accept (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-user-logging-forward (0 references)
pkts bytes target prot opt in out source destination
Chain ufw-user-logging-input (0 references)
pkts bytes target prot opt in out source destination
Chain ufw-user-logging-output (0 references)
pkts bytes target prot opt in out source destination
Chain ufw-user-output (1 references)
pkts bytes target prot opt in out source destination</redacted></redacted>
My typical approach is to capture tcpdump to a file (tcpdump -n -i eth0 -s 0 -w foo.pcap), let it run for awhile, then use wireshark to analyze it on a local computer.
Turns out it was a columbian IP address that was constantly accessing the site. I'm not sure exactly what they were doing yet (I'm still learning to interpret ntop info), but once I added their IP to the firewall the traffic IO graphs went back to normal (0 except for tiny bumps during cronjob emails to myself and whenever I access the site).
Thanks for your help guys!