bug after explain Linode
Recently, we've bought you linode 2048. Now, we have expanded our linode to 4096…
Linode, before expansion, was subjected to DDoS attacks (as well as now). To optimize server performance, we have done:
Used fail2ban utility, which monitored access.log our ngnix8 web-server, and adds entries in iptables if needs. Thus, already filtered about 15,000 hosts from attacker's botnet.
Increased possible limit open files from ~200,000 to 1,000,000. And also the limit of open files per user from 1000 to 32768 soft and 65536 hard links. Since the log file was included:
2010/12/10 01:22:13 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/10 01:22:13 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/10 01:22:13 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/10 01:22:13 [alert] 2097#0: accept () failed (1924: Too many open files)
In addition, we increased the number of workerconnections to 32768 connections, on our nginx8 web server. Because in the log file was wrote, that it is not enough workerconnections, after we increased the limit of open files.
All this has significantly reduced the size of error.log file, and decrease the speed of its completion, which accelerate the performance of the system.
And then today, we have expanded our Linode …
And now we see the same - error.log quickly filled with messages that:
2010/12/19 02:26:52 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/19 02:26:52 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/19 02:26:52 [alert] 2097#0: accept () failed (1924: Too many open files)
2010/12/19 02:26:52 [alert] 2097#0: accept () failed (1924: Too many open files)
Server performance - fell sharply.
We even increased the limits but it does not help.
Terminal's commands shows the following:
newmen@li199-4:/opt/nginx8/logs$ cat /proc/sys/fs/file-max
1000000
newmen@li199-4:/opt/nginx8/logs$ ulimit -Sn
65536
newmen@li199-4:/opt/nginx8/logs$ ulimit -Hn
131072
Why do not our old settings?
Please help me understand the issue.
Thanks a lot. Best wishes
Gleb
2 Replies
I reduced the limits of open files to the old values, and rebooted Linode several times…
And now everything works as before.
Thanks a lot.
Can't find the article @ the moment but I believe that *workerprocesses * workerconnections * 2* is a good rule of thumb (where worker_processes matches your available cores).