Excessive Traffic for odd ports like DNS

@craigw9292:

What I have noticed during a period of high traffic is 5 processes running perl under the www-data account - but not traffic on 80, seems to be on 53 again. I also can't kill those processes.

You mean you can't kill them or your services will not function, or the system won't let you kill them?

Post the results of ps aux – and use tcpdump to save some of the traffic on :53 -- tcpdump -i eth0 -s0 -w network-dump port 53

This will save a file named "network-dump" which you can open with wireshark to see what is going on.

Hmmm… try posting the output of these commands:

ps auxwww

netstat -nutaw

The first will produce a list of all processes running; the second will show all network connections currently open. There's a few possibilities for what's going on, and they usually aren't good.

@kbrantley:

@craigw9292:

What I have noticed during a period of high traffic is 5 processes running perl under the www-data account - but not traffic on 80, seems to be on 53 again. I also can't kill those processes.

You mean you can't kill them or your services will not function, or the system won't let you kill them?

Thanks for the help - I can't kill them - I issue a kill, they don't die. Each takes up about 20% cpu so the instance is maxed out. A reboot stops this for awhile … totally unpredictable when it will restart.

Posting all the dump info shortly.

-c

Here is the ps auxwww - fyi: running ubuntu 8.10 - upgraded.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.2 2012 952 ? Ss 10:25 0:00 /sbin/init

root 2 0.0 0.0 0 0 ? S 10:25 0:00 [migration/0]

root 3 0.0 0.0 0 0 ? SN 10:25 0:00 [ksoftirqd/0]

root 4 0.0 0.0 0 0 ? S 10:25 0:00 [migration/1]

root 5 0.0 0.0 0 0 ? SN 10:25 0:00 [ksoftirqd/1]

root 6 0.0 0.0 0 0 ? S 10:25 0:00 [migration/2]

root 7 0.0 0.0 0 0 ? SN 10:25 0:00 [ksoftirqd/2]

root 8 0.0 0.0 0 0 ? S 10:25 0:00 [migration/3]

root 9 0.0 0.0 0 0 ? SN 10:25 0:00 [ksoftirqd/3]

root 10 0.0 0.0 0 0 ? S< 10:25 0:00 [events/0]

root 11 0.0 0.0 0 0 ? S< 10:25 0:00 [events/1]

root 12 0.0 0.0 0 0 ? S< 10:25 0:00 [events/2]

root 13 0.0 0.0 0 0 ? S< 10:25 0:00 [events/3]

root 14 0.0 0.0 0 0 ? S< 10:25 0:00 [khelper]

root 15 0.0 0.0 0 0 ? S< 10:25 0:00 [kthread]

root 17 0.0 0.0 0 0 ? S< 10:25 0:00 [xenwatch]

root 18 0.0 0.0 0 0 ? S< 10:25 0:00 [xenbus]

root 27 0.0 0.0 0 0 ? S< 10:25 0:00 [kblockd/0]

root 28 0.0 0.0 0 0 ? S< 10:25 0:00 [kblockd/1]

root 29 0.0 0.0 0 0 ? S< 10:25 0:00 [kblockd/2]

root 30 0.0 0.0 0 0 ? S< 10:25 0:00 [kblockd/3]

root 31 0.0 0.0 0 0 ? S< 10:25 0:00 [cqueue/0]

root 32 0.0 0.0 0 0 ? S< 10:25 0:00 [cqueue/1]

root 33 0.0 0.0 0 0 ? S< 10:25 0:00 [cqueue/2]

root 34 0.0 0.0 0 0 ? S< 10:25 0:00 [cqueue/3]

root 36 0.0 0.0 0 0 ? S< 10:25 0:00 [kseriod]

root 116 0.0 0.0 0 0 ? S 10:25 0:00 [pdflush]

root 117 0.0 0.0 0 0 ? S 10:25 0:00 [pdflush]

root 118 0.0 0.0 0 0 ? S< 10:25 0:00 [kswapd0]

root 119 0.0 0.0 0 0 ? S< 10:25 0:00 [aio/0]

root 120 0.0 0.0 0 0 ? S< 10:25 0:00 [aio/1]

root 121 0.0 0.0 0 0 ? S< 10:25 0:00 [aio/2]

root 122 0.0 0.0 0 0 ? S< 10:25 0:00 [aio/3]

root 124 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsIO]

root 125 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsCommit]

root 126 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsCommit]

root 127 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsCommit]

root 128 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsCommit]

root 129 0.0 0.0 0 0 ? S< 10:25 0:00 [jfsSync]

root 130 0.0 0.0 0 0 ? S< 10:25 0:00 [xfslogd/0]

root 131 0.0 0.0 0 0 ? S< 10:25 0:00 [xfslogd/1]

root 132 0.0 0.0 0 0 ? S< 10:25 0:00 [xfslogd/2]

root 133 0.0 0.0 0 0 ? S< 10:25 0:00 [xfslogd/3]

root 134 0.0 0.0 0 0 ? S< 10:25 0:00 [xfsdatad/0]

root 135 0.0 0.0 0 0 ? S< 10:25 0:00 [xfsdatad/1]

root 136 0.0 0.0 0 0 ? S< 10:25 0:00 [xfsdatad/2]

root 137 0.0 0.0 0 0 ? S< 10:25 0:00 [xfsdatad/3]

root 746 0.0 0.0 0 0 ? S< 10:25 0:00 [net_accel/0]

root 747 0.0 0.0 0 0 ? S< 10:25 0:00 [net_accel/1]

root 748 0.0 0.0 0 0 ? S< 10:25 0:00 [net_accel/2]

root 749 0.0 0.0 0 0 ? S< 10:25 0:00 [net_accel/3]

root 756 0.0 0.0 0 0 ? S< 10:25 0:00 [kpsmoused]

root 759 0.0 0.0 0 0 ? S< 10:25 0:00 [kcryptd/0]

root 760 0.0 0.0 0 0 ? S< 10:25 0:00 [kcryptd/1]

root 761 0.0 0.0 0 0 ? S< 10:25 0:00 [kcryptd/2]

root 762 0.0 0.0 0 0 ? S< 10:25 0:00 [kcryptd/3]

root 763 0.0 0.0 0 0 ? S< 10:25 0:00 [kmirrord]

root 773 0.0 0.0 0 0 ? S< 10:25 0:00 [kjournald]

root 957 0.0 0.1 2188 480 ? S ~~root 2064 0.0 0.1 2140 432 ? Ss 10:25 0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0

root 2182 0.0 0.1 1680 500 tty1 Ss+ 10:25 0:00 /sbin/getty 38400 tty1

syslog 2214 0.0 0.1 1892 656 ? Ss 10:25 0:00 /sbin/syslogd -u syslog

root 2232 0.0 0.1 1832 532 ? S 10:25 0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg

klog 2234 0.0 0.2 2172 1036 ? Ss 10:25 0:00 /sbin/klogd -P /var/run/klogd/kmsg

root 2252 0.0 0.2 5276 868 ? Ss 10:25 0:00 /usr/sbin/sshd

ntop 2411 7.8 9.3 165248 34556 ? Ssl 10:25 21:04 /usr/sbin/ntop -d -L -u ntop -P /var/lib/ntop --access-log-file /var/log/ntop/access.log -i eth0 -p /etc/ntop/protocol.list -O /var/log/ntop

root 2493 0.0 0.4 5624 1684 ? Ss 10:25 0:00 /usr/lib/postfix/master

postfix 2500 0.0 0.4 5676 1624 ? S 10:25 0:00 qmgr -l -t fifo -u

nobody 2513 0.0 0.2 5268 844 ? Ss 10:26 0:00 proftpd: (accepting connections)

root 3214 0.0 0.2 2076 892 ? Ss 10:28 0:00 /usr/sbin/cron

clamav 3549 0.0 0.3 3136 1216 ? Ss 10:28 0:04 /usr/bin/freshclam -d --quiet

clamav 4665 0.0 0.1 88492 552 ? Ss 10:42 0:00 /usr/sbin/clamd

root 5552 0.0 0.3 2536 1192 ? S 10:48 0:00 /bin/sh /usr/bin/mysqld_safe

root 5703 0.0 3.9 50008 14548 ? Ss 10:48 0:00 /usr/sbin/apache2 -k start

root 5743 0.0 0.3 3532 1252 ? Sl 10:48 0:00 /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/ApplicationPoolServerExecutable 0 /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/bin/passenger-spawn-server /usr/bin/ruby1.8 /tmp/passenger_status.5703.fifo

root 5744 0.0 1.2 14252 4560 ? Sl 10:48 0:02 Passenger spawn server

mysql 5795 0.0 12.8 133192 47368 ? Sl 10:48 0:13 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

root 5796 0.0 0.1 1664 540 ? S 10:48 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld

www-data 5852 0.0 5.6 59440 20952 ? S 10:57 0:00 /usr/sbin/apache2 -k start

www-data 5892 0.0 0.0 0 0 ? Z 11:16 0:00 [sh] www-data 5894 0.0 0.7 4860 2856 ? S 11:16 0:00 /usr/bin/perl

postfix 6271 0.0 0.4 5632 1636 ? S 13:45 0:00 pickup -l -t fifo -u -c

www-data 6368 0.0 4.3 54404 15896 ? S 14:34 0:00 /usr/sbin/apache2 -k start

www-data 6372 0.0 4.6 55648 17264 ? S 14:34 0:01 /usr/sbin/apache2 -k start

www-data 6374 0.0 4.4 55100 16560 ? S 14:34 0:00 /usr/sbin/apache2 -k start

www-data 6384 0.0 4.5 55460 16812 ? S 14:35 0:00 /usr/sbin/apache2 -k start

www-data 6388 0.0 4.4 55436 16564 ? S 14:36 0:00 /usr/sbin/apache2 -k start

www-data 6390 0.0 4.1 54156 15468 ? S 14:38 0:00 /usr/sbin/apache2 -k start

www-data 6411 0.0 4.7 56384 17620 ? S 14:49 0:00 /usr/sbin/apache2 -k start

www-data 6412 0.0 2.3 50572 8672 ? S 14:50 0:00 /usr/sbin/apache2 -k start

www-data 6413 0.0 4.3 54588 15884 ? S 14:50 0:00 /usr/sbin/apache2 -k start

www-data 6418 0.0 4.3 54588 15880 ? S 14:52 0:00 /usr/sbin/apache2 -k start

root 6425 0.0 0.7 8068 2728 ? Ss 14:54 0:00 sshd: root@pts/0

root 6427 0.0 0.4 2884 1632 pts/0 Ss 14:54 0:00 -bash

www-data 6446 0.0 2.1 50440 7988 ? S 14:55 0:00 /usr/sbin/apache2 -k start

root 6447 0.0 0.2 2348 912 pts/0 R+ 14:55 0:00 ps auxwww~~

here is the other dump

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN

tcp 0 0 64.22.124.56:80 68.171.235.51:54338 ESTABLISHED

tcp 0 0 64.22.124.56:80 68.171.235.51:60709 TIME_WAIT

tcp 0 0 64.22.124.56:80 68.171.235.51:45924 TIME_WAIT

tcp 0 0 64.22.124.56:58564 91.121.14.55:8080 ESTABLISHED

tcp 0 0 64.22.124.56:80 68.171.235.51:45170 TIME_WAIT

tcp 0 0 64.22.124.56:80 68.171.235.51:54262 ESTABLISHED

tcp 0 0 64.22.124.56:80 68.171.235.51:58366 TIME_WAIT

tcp 0 48 64.22.124.56:22 76.20.230.161:49694 ESTABLISHED

tcp6 0 0 :::21 :::* LISTEN

tcp6 0 0 :::22 :::* LISTEN

tcp6 0 0 :::3000 :::* LISTEN

tcp6 0 0 :::25 :::* LISTEN

udp 0 0 0.0.0.0:68 0.0.0.0:*

here is the network dump during an attack on 53 - I blocked the IP of silverlords.org but looks like its still involved. Below is just a sample.

thanks

-c

‘√≤°

ok, so that didn't post -

here is a direct link to the dump

http://www.ip80.com/network-dump

You're requesting the same domain about 10 times a second. This is broken. If your host is the one in the dump, you should add it to /etc/hosts or run your own local DNS. However, that comes out to 8.7MB/hour of DNS traffic. While this is absurdly high (for your situation), I doubt it is the root cause of your issues.

But what you should really do is figure out what is causing those queries.

At first glance, your processes and your network connections look fine.

Try killall -9 <name of="" processes="" that="" are="" running="" perl=""> to see if that closes them out.

Also turn the logging up on your web server to see what is going on.

edit: from your network dump, it looks like you were hit by Google's web crawler at least once during that session. You should turn off reverse DNS lookups in apache.

This concerns me, from the netstat output:

tcp 0 0 64.22.124.56:58564 91.121.14.55:8080 ESTABLISHED

91.121.14.55 figures prominently in the tcpdump as well… is that IP known to you? Does your server have any reason to be contacting it on port 8080? If not, I suspect shenanigans.

On the netstat, I should have had you do:

netstat -nutawp

(Must be run as root)

That adds a column for the process that owns a connection:

rtucker@arrogant-bastard:~$ sudo netstat -nutawp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.1.12:51873      74.125.91.109:993       ESTABLISHED 30002/evolution 

In this case, I know process 30002 is connecting to 74.125.91.109:993 and – if I wasn't expecting that -- I'd know where to investigate further.

If you have lsof installed, by the way, you can do something like:

lsof -p30002

… and it'll tell you everything that pid 30002 has open. This will be a startlingly long list (it's 358 lines for that pid on my system!), but it can be valuable information when trying to figure out what, exactly, is going on.

Great thanks - there is no reason for my server to have an outbound connection to anyone on 8080.

I don't see it anymore on nutawp so I'll wait for it to return and run lsof on 8080. I did put the domain that was repeating in my dns requests into my host file to stop connections - pointed it to localhost instead, that might have helped for now.

-c

Probably been exploited somehow with an RFI or some sort and droped a crappy perl IRC bot onto your system being used to DoS pepole or scan for other vulnerable boxes.