memcached taking all of my ram?

Hey all,

I setup a 768 linode and host 1 site on it. It's a fairly typical site - blog-type content. I'm using MySQL, nginx, php-fpm and memcached.

So, I had originally set it up without memcached and right around 1500 visitors an hour, it started using a ton of disk IO.

  • The site calls about 5 sql queries for each page render.

  • I have nginx setup with 4 workers and 1024 worker connections

  • php is config'd to use 6mb of ram per thread

  • ulimit -n is set to 1024

  • The VPS has 768mb of ram

To bring the IO usage down, I installed memcached and started setting up the queries to use it and cache queries for 10 minutes. Seemed to work great. Fast load times still and the IO usage went way down… until about 10 hours later when I was getting nginx's gateway timeout's. Checking the logs and free -m told me that it was out of memory. I restarted memcached and it's all fine and dandy again. Even with memcached, it locks up around the 1500/user/hour threshold.

These are my settings for memcached:

MAXCONN="1024"

CACHESIZE="64"

Here's the seemingly important bits of my /var/log/messages right before it locks up:

Mar 15 01:42:00 host kernel: HighMem free:208kB min:128kB low:208kB high:288kB activeanon:25520kB inactiveanon:25632kB active_file:480kB inact$

Mar 15 01:42:00 host kernel: lowmem_reserve[]: 0 0 0 0

Mar 15 01:42:00 host kernel: DMA: 984kB 198kB 316kB 132kB 164kB 1128kB 2256kB 1512kB 11024kB 02048kB 0*4096kB = 2864kB

Mar 15 01:42:00 host kernel: Normal: 13584kB 28kB 116kB 132kB 064kB 0128kB 2256kB 0512kB 11024kB 02048kB 0*4096kB = 7032kB

Mar 15 01:42:00 host kernel: HighMem: 264kB 58kB 016kB 032kB 164kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 208kB

Mar 15 01:42:00 host kernel: 1592 total pagecache pages

Mar 15 01:42:00 host kernel: 126 pages in swap cache

Mar 15 01:42:00 host kernel: Swap cache stats: add 4286470, delete 4286344, find 616656/770806

Mar 15 01:42:00 host kernel: Free swap = 0kB

Mar 15 01:42:00 host kernel: Total swap = 1048572kB

snip

Out of memory: Kill process ###

Is there any way to detect if it's about to run out of memory and flush memcached or restart it? Or am I going about this in the wrong way somehow?

4 Replies

Try 'ps axl' to confirm that the problem is memcached eating all the ram.

First: You probably don't want your swap to be 1 GB. Try lowering it to 256 MB or 128 MB, so that your system can take action against memory hogs before everything grinds to a halt.

Second: It's probably not memcached. You're probably swap-thrashing, which is probably due to either PHP running amok or bad MySQL settings. Try running mysqltuner.pl and posting what it reports. Also, your pool config for php-fpm might prove helpful.

So, it seemingly turned out to be php-fpm and some gratuitous default settings. I lowered the maxchildren to 30 (from 50) and lowered startservers and maxservers to 5, with maxspareservers set to 8, with maxrequests set to 50. I also raised the OS' ulimit to around 4000. Seems to be solid at the moment with similar traffic. I also did a test from loadimpact.com and pushed 2000 simultaneous "virtual" users at it. It hovered around 10% CPU and still had about 400Mb of RAM free during that time. Thanks folks :)

max_requests of 50 is really low. Theoretically you shouldn't need it. Setting it to something in the 1000s would be better.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct