PHP/MySQL/Apache tuning advice
Problem: 3 times in the past 2 days the site stopped responding. CPU was idle, 1.8gb RAM free. Nothing interesting in the Apache, MySQL, or PHP logs. Restarting Apache and MySQL did not help. Rebooting fixed the problem for ~14 hours. syslog shows many, many of these:
localhost kernel: nf_conntrack: table full, dropping packet
localhost kernel: nf_conntrack: table full, dropping packet
localhost kernel: net_ratelimit: 8 callbacks suppressed
localhost kernel: nf_conntrack: table full, dropping packet
After some Google-fu, I found this informative page:
Increasing nfconntrackmax allowed the site to respond again, but it is possible that whatever the problem was will just cause it to hit the max again. Still, I edited /etc/sysctl.conf and added:
net.netfilter.nf_conntrack_max = 32768
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
I also found this in my sysctl.conf:
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000
I'm guessing I put this in on some previous tuning a long time ago. I'm not sure if it is any good?
Linode support (those guys are great!) suggested leaving nfconntrackmax at the default (16384). One theory is something is up with my iptables, possibly "some odd circular iptables entries". I had used arno's firewall script and also installed fail2ban. Here's my iptables:
Very sorry for the formatting, that was copied out of the Lish console while the site was not responding. I don't really know anything about iptables, so I don't know if all that stuff is reasonable. Support suggested rebooting then doing this to reset the iptables:
service fail2ban stop
service arno-iptables-firewall stop
iptables -F
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
So I did that and the site is running fine, though fully open. I can give it a few days to see if it stops responding. Assuming the site is ok with the iptables reset, what do I do for a permanent solution? I don't want to bother Linode support further, since it's clearly not a problem on their end, so I was hoping maybe someone here could help.
Support mentioned, "If it is working properly, then my guess is that it's arno's script. There's some configuration you can do with it but honestly I haven't seen a case where the script performs better than fail2ban for typical use." Does this mean I should not have used both arno and fail2ban? Is fail2ban alone enough to protect me? All I did with arno is open 6 ports, so I'm not doing anything fancy. I installed fail2ban because I thought it might protect from HTTP basic auth and SSH brute force.
4 Replies
You don't need any of this fail2ban/arno's script craziness.
You can allow SSH, web, NTP, DNS from anywhere, anything from 127/8, and block everything else. Once that works you may figure out what else needs to send or receive packets and add rules for that. Longview will help if you are not already using it, as will 'ss -nptl' or 'netstat -nptl'.
@sednet:
Setup good passwords for apache and only use public key authentication for SSH.
You don't need any of this fail2ban/arno's script craziness.
You can allow SSH, web, NTP, DNS from anywhere
Don't forget SMTP, too, if you're running e-mail for your site on the same node.
Got Longview set up, really nice!
I kept fail2ban and redid my iptables manually, here's my script (no SMTP):
I read up on sysctl settings and tried to pick values that meet my needs:
Any feedback on these? What do you guys do for a 2048 Linode that is just PHP/MySQL/Apache?
I guess that is all I can do. I'll just have to give it some time and see how it goes. I signed up for Linode Managed, so at least they can reboot it when I'm asleep (it only hangs when I'm asleep…) and my users won't be too affected.
Also, here are my Apache settings:
~~![](<URL url=)http://i.imgur.com/6wtC7ep.png
Not sure why that happened, but the machine handled it. Seems it was mostly the Apache processes, here is a graph of the workers just at that time:
~~![](<URL url=)http://i.imgur.com/DlK2eET.png
More charts showing requests Apache receive jumping from 1/s to about 3/s:
~~![](<URL url=)http://i.imgur.com/bFq4TLp.png
Guess this is fine? I have lots of free memory, do you think I should be using it? Using ab to bench I think Apache will use ~1gb max.
One more chart to show MySQL:
~~![](<URL url=)http://i.imgur.com/5ARvkFe.png
MySQL doesn't seem taxed at all. Here's my config:
I haven't read about it much, so I'm not sure this is any good. AFAIK all my tables are MyISAM. Is the keybuffer under [isamchk] overwriting keybuffer_size set earlier?
Any feedback on my setup is highly appreciated! ~~~~