Spike in CPU and outgoing data

Here are the graphs for the cpu, bandwidth, and disk io.

~~http://i.imgur.com/eGqMdZo.png" />

~~http://i.imgur.com/tRX8job.png" />

![](" />~~~~

Personally I'd want to know what was going on on my own Linode. That type of out-of-the-ordinary activity usually means something needs your attention (such as a breach, infection, etc) or that some bot was scraping everything from your site.

What distro, what type of site(s), what do your logs say for that time period, etc?

distro: Ubuntu 10.04.4 LTS

sites: 2 sites with little to no traffic

tech stack: ruby on rails, passenger, apache, git, capistrano

i have disabled root login, and have fail2ban installed.

auth.log previously showed many attempts to log in with non-existing users from IPs that have been flagged for dirty activity. then i installed fail2ban. now there seem to be fewer entries in the logs, but still significant auth attempts.

auth.log:

Jun  8 22:56:02 localhost sshd[12241]: Failed none for invalid user ubnt from 191.238.36.164 port 1080 
ssh2
Jun  8 22:56:02 localhost sshd[12241]: pam_unix(sshd:auth): check pass; user unknown
Jun  8 22:56:04 localhost sshd[12241]: Failed password for invalid user ubnt from 191.238.36.164 port 1
080 ssh2

access log shows some cgi-bin requests, which i don't think have anything to do with the rails framework.

access.log:

root@li681-185:~# grep "09/Jun/2014:10" /var/log/apache2/life_catalog_access.log
1.214.212.74 - - [09/Jun/2014:10:00:30 -0400] "GET //cgi-bin/php HTTP/1.1" 404 974 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
1.214.212.74 - - [09/Jun/2014:10:00:31 -0400] "GET //cgi-bin/php5 HTTP/1.1" 404 974 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
1.214.212.74 - - [09/Jun/2014:10:00:31 -0400] "GET //cgi-bin/php-cgi HTTP/1.1" 404 974 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
1.214.212.74 - - [09/Jun/2014:10:00:32 -0400] "GET //cgi-bin/php.cgi HTTP/1.1" 404 974 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
1.214.212.74 - - [09/Jun/2014:10:00:33 -0400] "GET //cgi-bin/php4 HTTP/1.1" 404 974 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"

I'm also seeing this when I run who:

kenstclair@li681-185:~$ who -H

NAME     LINE         TIME             COMMENT
kenstclair hvc0         2014-05-24 14:32
kenstclair pts/2        2014-06-09 17:24 (cpe-68-173-79-93.nyc.res.rr.com)

a search for hvc0 shows links to a hypervisor console (http://unix.stackexchange.com/questions … n-who-list">http://unix.stackexchange.com/questions/10857/what-is-hvc0-appearing-in-who-list) which I am not using.

You used it 20 days ago, and didn't logout properly; just disconnected from LISH

I am not very optimistic, given that all of the spinning processes are running as the same non-privileged username (i.e. yours), have identical memory usage, and all started at almost exactly the same time. I'd say your user got pwned.

@kenstclair:

I just received a couple of notices about my CPU and bandwidth usage, and am trying to figure out what is causing them, and if there might be anything malicious going on.

> Your Linode, linodexxx, has exceeded the notification threshold (5) for outbound traffic rate by averaging 5.67 Mb/s for the last 2 hours.

Your Linode, linodexxxx, has exceeded the notification threshold (90) for CPU Usage by averaging 155.1% for the last 2 hours.

Logging in, I see a spike in the CPU usage and outgoing data, as noted. CPU usage has been up all morning, and there were 5GB of outgoing data, up from nothing.

Top shows the following:

$ top
top - 11:32:45 up 57 days,  9:44,  3 users,  load average: 15.02, 14.94, 14.98
Tasks: 148 total,  16 running, 132 sleeping,   0 stopped,   0 zombie
Cpu(s): 28.5%us, 10.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 61.0%st
Mem:   1026840k total,   930096k used,    96744k free,   114824k buffers
Swap:   262140k total,    12280k used,   249860k free,   506520k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                          
23945 kenstcla  20   0  6624 3048 1316 R   66  0.3 148:43.23 /usr/sbin/acpid                                                                   
23992 kenstcla  20   0  6624 3044 1316 R   59  0.3 149:21.61 /usr/local/apache/bin/httpd -DSSL                                                 
23957 kenstcla  20   0  6624 3044 1316 R   58  0.3 151:28.32 /usr/sbin/acpid                                                                   
23952 kenstcla  20   0  6624 3048 1316 R   55  0.3 150:22.51 /sbin/klogd -c 1 -x -x                                                            
23985 kenstcla  20   0  6624 3044 1316 R   55  0.3 150:26.12 /sbin/klogd -c 1 -x -x                                                            
23923 kenstcla  20   0  6624 3044 1312 R   53  0.3 152:20.83 /usr/sbin/sshd -i                                                                 
23940 kenstcla  20   0  6624 3048 1316 R   53  0.3 149:21.66 /usr/sbin/httpd                                                                   
23972 kenstcla  20   0  6624 3044 1316 R   53  0.3 151:24.26 /usr/sbin/acpid                                                                   
23997 kenstcla  20   0  6624 3048 1316 R   52  0.3 148:53.54 /usr/sbin/cron                                                                    
23935 kenstcla  20   0  6624 3044 1312 R   51  0.3 152:23.89 /usr/sbin/sshd -D                                                                 
24002 kenstcla  20   0  6624 3044 1316 R   50  0.3 149:11.03 /usr/sbin/cron                                                                    
23930 kenstcla  20   0  6624 3040 1312 R   50  0.3 152:16.68 /usr/local/apache/bin/httpd -DSSL                                                 
23962 kenstcla  20   0  6624 3048 1316 R   50  0.3 148:40.71 /usr/sbin/sshd -D                                                                 
23967 kenstcla  20   0  6624 3048 1316 R   50  0.3 151:17.99 /sbin/syslogd                                                                     
23977 kenstcla  20   0  6624 3044 1316 R   49  0.3 150:28.61 /usr/sbin/sshd                                                                    
    7 root      20   0     0    0    0 S    3  0.0  15:26.26 [rcu_sched]                                                                       
24128 root      20   0 32724 3312 2680 S    1  0.3   1:30.92 PassengerHelperAgent                                                              
30991 root      20   0  2632 1124  832 R    0  0.1   0:10.73 top                                                                               
    1 root      20   0  2868 1412 1116 S    0  0.1   0:14.63 /sbin/init                                                                        
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 [kthreadd]                                                                        
    3 root      20   0     0    0    0 S    0  0.0   0:06.81 [ksoftirqd/0]       

Should I be worried about these?

I'd be worried with your normal user account password. No matter what the situation - there is no valid reason for a user account to be running stuff like syslogd, klogd, acpid and cron.

hoopycat's right about the procs having same footprint being a really suspicious thing.

Now's a good time to validate your backups. Because I'd login as root, copy really important data out - then nuke from orbit.

If you want you can also login as root and put down the network interface (eth0), then kill all of these rouge processes.

@ken-ji:

Because I'd login as root, copy really important data out - then nuke from orbit.

If you want you can also login as root and put down the network interface (eth0), then kill all of these rouge processes.

Rebooting into rescue mode is safer.

I rebooted the linode and changed my user's password. Haven't seen any activity since then.

@kenstclair:

I rebooted the linode and changed my user's password. Haven't seen any activity since then.
And you think that's it?

You have no clue what happened, how it happened, what was changed/installed, but you changed a user password and you think it's all shiny happy now?

I have a new powdered water product getting ready to hit the market and am looking for investors - please call me.