CPU and outgoing traffic WAY to high
Please try to help me before my client helps him self with a sharp knife.
I'm on a linode512. Having Apache/PHP/MySql/Wordpress site with an ftp server (turned off) and postfix/dovecot combination.
Please look at what is going on with my server graphs the last few hours:
~~http://i41.tinypic.com/10gzb6a.png
As you can see I've even tried to restart the server after restarting httpd & mysqld (yeah I know it's stupid, but I had a client yelling over the phone so I was looking for a quick fix)… obviously it didn't help.
netstat -an | grep :80
gave me some unusual number of open connections, so I became suspicious of a DOS or DDOS attack and started googling. I found the following command which should give me the number of connections per IP address:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
This is the output:
1 146.255.65.245
1 213.149.150.251
1 31.11.72.38
1 31.11.73.110
1 31.11.84.175
1 46.217.69.242
1 46.217.94.33
1 77.199.39.207
1 77.28.13.29
1 77.29.134.127
1 77.29.150.201
1 77.29.201.128
1 77.29.5.110
1 79.125.129.119
1 79.125.139.83
1 79.125.148.229
1 79.125.169.178
1 79.125.179.183
1 79.125.193.7
1 79.126.132.200
1 79.126.193.104
1 79.126.243.230
1 79.126.251.213
1 89.205.7.113
1 89.205.7.72
1 92.53.0.195
1 92.53.15.71
1 95.156.52.17
1 95.86.52.159
2 195.26.144.127
2 77.28.12.63
2 77.28.156.97
2 77.28.28.247
2 77.29.173.157
2 77.29.79.7
2 78.157.2.114
2 78.157.27.0
2 79.125.197.222
2 79.125.234.208
2 79.126.208.111
2 79.126.214.155
2 79.126.217.97
2 89.185.195.212
2 89.185.221.229
2 89.205.14.161
3 151.67.110.91
3 31.11.79.4
3 46.217.116.213
3 46.217.3.153
3 62.162.205.250
3 77.28.35.111
3 77.28.98.134
3 77.29.216.94
3 77.29.51.90
3 89.205.42.167
3 92.55.104.182
4 46.217.12.13
4 77.28.149.35
4 79.141.120.118
4 87.4.65.169
4 92.53.62.25
5 77.28.114.120
5 77.29.208.192
5 79.141.125.231
6 78.157.17.3
6 79.125.198.179
15 0.0.0.0
1816
Would you look at that LAST row. What is that number ?!?
Can I find out what that is??
Now, I know I can optimize the apache & mysql configuration up until a certain point. This is my current configuration which worked great up until what happend few hours ago:
thread_cache_size = 4
tmp_table_size = 32M
max_heap_table_size = 32M
max_write_lock_count = 1
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 16M
<ifmodule prefork.c="">StartServers 3
MinSpareServers 3
MaxSpareServers 8
ServerLimit 12
MaxClients 12
MaxRequestsPerChild 1000</ifmodule>
I'm also logging slow MySQL queries for an hour now, and the file is still empty…
So at this point I really don't know what to do. And what are those 1816 which are apparently coming from nowhere…
Please….. help…………..~~
13 Replies
netstat -anp | grep ':::'
and see what processes use those ports.
tcp 0 0 :::993 :::* LISTEN 1888/dovecot
tcp 0 0 :::995 :::* LISTEN 1888/dovecot
tcp 0 0 :::110 :::* LISTEN 1888/dovecot
tcp 0 0 :::143 :::* LISTEN 1888/dovecot
tcp 0 0 :::80 :::* LISTEN 7594/httpd
tcp 0 0 :::22 :::* LISTEN 1714/sshd
tcp 0 0 :::25 :::* LISTEN 1968/master
tcp 0 0 :::443 :::* LISTEN 7594/httpd
udp 0 0 ::1:123 :::* 1722/ntpd
udp 0 0 fe80::f03c:91ff:fedf:195:123 :::* 1722/ntpd
udp 0 0 :::123 :::* 1722/ntpd
Use "ps axu" to check what process/es is/are using CPU (you can paste output here)
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2924 936 ? Ss 13:21 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 13:21 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/u:0]
root 6 0.0 0.0 0 0 ? S 13:21 0:00 [migration/0]
root 7 0.0 0.0 0 0 ? S 13:21 0:00 [migration/1]
root 8 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/1:0]
root 9 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S 13:21 0:00 [migration/2]
root 11 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/2:0]
root 12 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/2]
root 13 0.0 0.0 0 0 ? S 13:21 0:00 [migration/3]
root 14 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/3:0]
root 15 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/3]
root 16 0.0 0.0 0 0 ? S< 13:21 0:00 [khelper]
root 17 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/u:1]
root 21 0.0 0.0 0 0 ? S 13:21 0:00 [xenwatch]
root 22 0.0 0.0 0 0 ? S 13:21 0:00 [xenbus]
root 148 0.0 0.0 0 0 ? S 13:21 0:00 [sync_supers]
root 150 0.0 0.0 0 0 ? S 13:21 0:00 [bdi-default]
root 152 0.0 0.0 0 0 ? S< 13:21 0:00 [kblockd]
root 162 0.0 0.0 0 0 ? S< 13:21 0:00 [md]
root 246 0.0 0.0 0 0 ? S< 13:21 0:00 [rpciod]
root 247 0.0 0.0 0 0 ? S 13:21 0:01 [kworker/0:1]
root 279 0.0 0.0 0 0 ? S 13:21 0:02 [kswapd0]
root 280 0.0 0.0 0 0 ? SN 13:21 0:00 [ksmd]
root 281 0.0 0.0 0 0 ? S 13:21 0:00 [fsnotify_mark]
root 285 0.0 0.0 0 0 ? S 13:21 0:00 [ecryptfs-kthrea]
root 287 0.0 0.0 0 0 ? S< 13:21 0:00 [nfsiod]
root 290 0.0 0.0 0 0 ? S 13:21 0:00 [jfsIO]
root 291 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 292 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 293 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 294 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 295 0.0 0.0 0 0 ? S 13:21 0:00 [jfsSync]
root 296 0.0 0.0 0 0 ? S< 13:21 0:00 [xfs_mru_cache]
root 297 0.0 0.0 0 0 ? S< 13:21 0:00 [xfslogd]
root 298 0.0 0.0 0 0 ? S< 13:21 0:00 [xfsdatad]
root 299 0.0 0.0 0 0 ? S< 13:21 0:00 [xfsconvertd]
root 300 0.0 0.0 0 0 ? S< 13:21 0:00 [glock_workqueue]
root 301 0.0 0.0 0 0 ? S< 13:21 0:00 [delete_workqueu]
root 302 0.0 0.0 0 0 ? S< 13:21 0:00 [gfs_recovery]
root 303 0.0 0.0 0 0 ? S< 13:21 0:00 [crypto]
root 865 0.0 0.0 0 0 ? S 13:21 0:00 [khvcd]
root 979 0.0 0.0 0 0 ? S< 13:21 0:00 [kpsmoused]
root 1002 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/2:1]
root 1007 0.0 0.0 0 0 ? S 13:21 0:01 [kjournald]
root 1029 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/1:1]
root 1036 0.0 0.0 0 0 ? S 13:21 0:00 [kauditd]
root 1045 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/3:1]
root 1081 0.0 0.0 2656 248 ? S ~~root 11828 0.0 0.2 4928 1044 pts/0 R+ 16:27 0:00 ps axu~~
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1 46.217.108.131
1 79.125.129.160
3 77.28.98.134
15 0.0.0.0
2238
2238?
And the processor is above 250%
cut -d: -f1
do
sed 's/:[0-9][0-9]*$//'
1 46.217.108.131
1 77.28.80.168
1 84.175.200.44
1 ::ffff:69.171.228.246
1 ::ffff:69.171.228.250
1 ::ffff:69.171.229.251
1 ::ffff:77.28.31.60
1 ::ffff:77.29.141.155
1 ::ffff:77.29.156.44
1 ::ffff:77.29.255.138
1 ::ffff:77.29.33.213
1 ::ffff:78.157.19.142
1 ::ffff:79.125.187.153
1 ::ffff:84.175.200.44
1 ::ffff:95.86.37.239
2 77.28.199.45
2 79.125.173.61
2 ::ffff:46.217.116.213
3 77.28.98.134
3 ::ffff:66.249.72.228
6 41.194.70.130
6 95.86.41.237
7 ::ffff:79.125.140.247
8 ::ffff:31.11.116.176
8 ::ffff:89.205.63.199
8 ::ffff:89.205.80.233
10 ::ffff:31.11.82.141
10 ::ffff:79.125.173.61
11 :::*
11 ::ffff:89.205.15.34
15 0.0.0.0:*
15 ::ffff:79.125.139.83
15 ::ffff:79.125.188.235
15 ::ffff:89.185.220.148
15 ::ffff:92.53.29.60
16 ::ffff:95.86.41.237
20 ::ffff:77.29.37.201
23 ::ffff:77.28.24.51
27 ::ffff:79.126.252.115
33 ::ffff:78.157.26.201
34 ::ffff:77.28.75.254
34 ::ffff:77.29.153.251
37 ::ffff:41.194.70.130
38 ::ffff:77.28.199.45
43 ::ffff:46.217.88.61
45 ::ffff:79.126.255.155
45 ::ffff:85.30.96.3
45 ::ffff:95.86.24.33
47 ::ffff:92.53.28.211
48 ::ffff:78.157.28.125
50 ::ffff:178.249.174.8
50 ::ffff:77.29.106.91
50 ::ffff:92.53.51.163
51 ::ffff:77.29.246.86
51 ::ffff:77.29.68.68
52 ::ffff:78.157.1.56
53 ::ffff:46.217.15.183
53 ::ffff:77.28.108.122
54 ::ffff:77.28.69.86
54 ::ffff:77.28.75.12
55 ::ffff:77.28.144.188
55 ::ffff:77.28.76.104
55 ::ffff:77.28.78.105
55 ::ffff:77.28.80.168
55 ::ffff:77.29.1.107
55 ::ffff:79.125.179.113
55 ::ffff:79.126.248.89
55 ::ffff:92.53.30.229
55 ::ffff:95.180.226.228
56 ::ffff:62.162.52.222
56 ::ffff:77.28.166.225
56 ::ffff:77.28.49.207
56 ::ffff:77.29.67.248
56 ::ffff:77.29.88.169
56 ::ffff:79.125.236.200
56 ::ffff:79.126.244.243
56 ::ffff:89.205.95.15
56 ::ffff:95.180.186.251
57 ::ffff:46.217.69.172
57 ::ffff:79.125.251.232
57 ::ffff:92.55.99.216
60 ::ffff:89.185.214.17
61 ::ffff:92.53.16.37
90 ::ffff:89.205.30.213
````
Is this a DDOS ???
You're using some sort of caching plugin for WordPress, right? MySQL seems to be grinding more than it should.
@obs:
Also have a go at running
http://mysqltuner.pl/mysqltuner.pl to help optimise mysql
That's how I got the configuration posted above. Here is what the script is advizing me:
-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 55m 27s (1M q [220.752 qps], 19K conn, TX: 3B, RX: 185M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 56.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 461.8M (92% of installed RAM)
[OK] Slow queries: 0% (0/1M)
[OK] Highest usage of available connections: 12% (19/151)
[OK] Key buffer size / total MyISAM indexes: 8.0M/4.6M
[OK] Key buffer hit rate: 99.6% (4M cached / 17K reads)
[OK] Query cache efficiency: 93.6% (1M cached / 1M selects)
[!!] Query cache prunes per day: 60768
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 15K sorts)
[!!] Temporary tables created on disk: 43% (4K on disk / 10K total)
[OK] Thread cache hit rate: 87% (2K created / 19K connections)
[OK] Table cache hit rate: 31% (56 open / 177 opened)
[OK] Open file limit used: 8% (88/1K)
[!!] Table locks acquired immediately: 94%
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
query_cache_size (> 16M)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
Now for what is worth, I did not have any caching mechanism installed. Now I have W3 Total Cache.
This is what happend after I deployed the caching configuration:
~~http://i41.tinypic.com/23hnjvo.png
But to be honest I am not sure which really helped?
Is it the plugin? Or is it the inbound trafic which also went down…?
I remember having only 4-5 connected users no more then two hours ago and the CPU was well over 200%…~~
Grab Apache logs from the heavy-traffic period. See if there's a set of specific URLs that most of the requests are pointing at. Also check for a referer, since this could have been caused by a popular link on Digg/Slashdot/Reddit/etc. If there's an easily recognized pattern, you can tell your client what really happened.
Next step, OPTIMIZE EVERYTHING. Install APC. Install a caching plugin (which you already did) and make it use the most aggressive caching method available. Consider switching PHP to FastCGI and/or changing your web server to nginx so that requests for static files don't clog up your server. Tune MySQL. Get a bigger linode if mysqltuner keeps complaining that there isn't enough RAM.
I've considered the last advice and did the following:
removed eaccelerator
installed APC
installed nginx as a proxy above the apache server
installed w3 total cache with page cache via opcode, minify via opcode, object cache via opcode and browser cache too.
…and I'll be waiting for some 20 more hours to see what will the mysqltune.pl script advise but I can see that prunes are already down to 0 and before they were 60000+
From the first look of it the site feels considerably faster.
I'll be waiting for the next busy period to determine how will the configuration cope with the traffic and post the results here.
Again, thank you very much!
170 concurent connections from 40 different IP addresses at the time this screenshot was taken.
![](