memcached taking all of my ram?
I setup a 768 linode and host 1 site on it. It's a fairly typical site - blog-type content. I'm using MySQL, nginx, php-fpm and memcached.
So, I had originally set it up without memcached and right around 1500 visitors an hour, it started using a ton of disk IO.
The site calls about 5 sql queries for each page render.
I have nginx setup with 4 workers and 1024 worker connections
php is config'd to use 6mb of ram per thread
ulimit -n is set to 1024
The VPS has 768mb of ram
To bring the IO usage down, I installed memcached and started setting up the queries to use it and cache queries for 10 minutes. Seemed to work great. Fast load times still and the IO usage went way down… until about 10 hours later when I was getting nginx's gateway timeout's. Checking the logs and free -m told me that it was out of memory. I restarted memcached and it's all fine and dandy again. Even with memcached, it locks up around the 1500/user/hour threshold.
These are my settings for memcached:
MAXCONN="1024"
CACHESIZE="64"
Here's the seemingly important bits of my /var/log/messages right before it locks up:
Mar 15 01:42:00 host kernel: HighMem free:208kB min:128kB low:208kB high:288kB activeanon:25520kB inactiveanon:25632kB active_file:480kB inact$
Mar 15 01:42:00 host kernel: lowmem_reserve[]: 0 0 0 0
Mar 15 01:42:00 host kernel: DMA: 984kB 198kB 316kB 132kB 164kB 1128kB 2256kB 1512kB 11024kB 02048kB 0*4096kB = 2864kB
Mar 15 01:42:00 host kernel: Normal: 13584kB 28kB 116kB 132kB 064kB 0128kB 2256kB 0512kB 11024kB 02048kB 0*4096kB = 7032kB
Mar 15 01:42:00 host kernel: HighMem: 264kB 58kB 016kB 032kB 164kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 208kB
Mar 15 01:42:00 host kernel: 1592 total pagecache pages
Mar 15 01:42:00 host kernel: 126 pages in swap cache
Mar 15 01:42:00 host kernel: Swap cache stats: add 4286470, delete 4286344, find 616656/770806
Mar 15 01:42:00 host kernel: Free swap = 0kB
Mar 15 01:42:00 host kernel: Total swap = 1048572kB
snip
Out of memory: Kill process ###
Is there any way to detect if it's about to run out of memory and flush memcached or restart it? Or am I going about this in the wrong way somehow?
4 Replies
Second: It's probably not memcached. You're probably swap-thrashing, which is probably due to either PHP running amok or bad MySQL settings. Try running mysqltuner.pl