View All Products
Compute
Storage
Networking
Databases
Services
Developer Tools
Industries
Pricing
Community
Engage With Us
Community ▶ Questions ▶ How to identify what caused high load and locked my linode?
Log in to Ask a Question

How to identify what caused high load and locked my linode?

general

Just noticed my linode was down, actually, the Wordpress site I'm hosting there. Rebooted it via linode manager.

Trying to figure out what happened. Can't see anything strange in the logs.

Here are graphs:

http://tinypic.com/r/11htkqr/7

http://tinypic.com/r/or6mf8/7

http://tinypic.com/r/2weanat/7

Everything started approximatelly @ 9:10pm as you may see on graphs. Few mins later server hanged and stopped responding.

Any idea where to take a look? Apache log looks good, mysqld log looks good too, nothing strange in secure and maillog.

Thanks

Josh

4 Replies

Found this in /var/log/messages:

Nov 27 04:02:04 liXXX-XXX syslogd 1.4.1: restart.

Nov 27 21:14:37 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20

Nov 27 21:14:37 liXXX-XXX kernel: Pid: 19979, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:14:37 liXXX-XXX kernel: Call Trace:

Nov 27 21:14:37 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x584/0x670

Nov 27 21:14:37 liXXX-XXX kernel: [] ? cacheallocrefill+0x2e3/0x5c0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? kmemcachealloc+0xa7/0xb0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? skprotalloc+0x42/0x130

Nov 27 21:14:37 liXXX-XXX kernel: [] ? sk_clone+0x14/0x290

Nov 27 21:14:37 liXXX-XXX kernel: [] ? inetcskclone+0x10/0xb0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpcreateopenreq_child+0x21/0x470

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpv4synrecvsock+0x53/0x1d0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpv6synrecvsock+0x46d/0x580

Nov 27 21:14:37 liXXX-XXX kernel: [] ? rawspin_lock+0x5/0x10

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpcheckreq+0x31e/0x4a0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpv4do_rcv+0x1c7/0x2a0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? localbhenable+0x22/0xb0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? tcpv4rcv+0x73b/0x8a0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? iplocaldeliver_finish+0xcb/0x200

Nov 27 21:14:37 liXXX-XXX kernel: [] ? iprcvfinish+0x191/0x3b0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? ip_rcv+0x256/0x2e0

Nov 27 21:14:37 liXXX-XXX kernel: [] ? netifreceiveskb+0x78/0x90

Nov 27 21:14:37 liXXX-XXX kernel: [] ? xennet_poll+0x59d/0xe20

Nov 27 21:14:37 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:14:37 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:14:37 liXXX-XXX kernel: [] ? netrxaction+0x97/0x1b0

Nov 27 21:14:38 liXXX-XXX kernel: [] ? _dosoftirq+0xae/0x140

Nov 27 21:14:38 liXXX-XXX kernel: [] ? unmask_evtchn+0x1a/0xf0

Nov 27 21:14:38 liXXX-XXX kernel: [] ? call_softirq+0x1c/0x30

Nov 27 21:14:38 liXXX-XXX kernel: [] ? do_softirq+0x65/0xa0

Nov 27 21:14:38 liXXX-XXX kernel: [] ? irq_exit+0x75/0x90

Nov 27 21:14:38 liXXX-XXX kernel: [] ? xenevtchndo_upcall+0x13e/0x200

Nov 27 21:14:38 liXXX-XXX kernel: [] ? xendohypervisor_callback+0x1e/0x30

Nov 27 21:14:38 liXXX-XXX kernel: [] ? ptepclearflush_young+0x35/0x80

Nov 27 21:14:38 liXXX-XXX kernel: [] ? pagereferencedone+0xc1/0x110

Nov 27 21:14:38 liXXX-XXX kernel: [] ? page_referenced+0x20f/0x230

Nov 27 21:14:38 liXXX-XXX kernel: [] ? shrinkpagelist+0x197/0x750

Nov 27 21:14:38 liXXX-XXX kernel: [] ? shrinkactivelist+0x25a/0x2d0

Nov 27 21:14:38 liXXX-XXX kernel: [] ? isolatelrupages+0x8c/0x220

Nov 27 21:14:39 liXXX-XXX kernel: [] ? shrink_zone+0x64b/0xc80

Nov 27 21:14:39 liXXX-XXX kernel: [] ? ioscheduletimeout+0xaf/0xf0

Nov 27 21:14:40 liXXX-XXX kernel: [] ? autoremovewakefunction+0x0/0x30

Nov 27 21:14:40 liXXX-XXX kernel: [] ? trytofree_pages+0x1dd/0x3a0

Nov 27 21:14:40 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x3b5/0x670

Nov 27 21:14:41 liXXX-XXX kernel: [] ? handlemmfault+0x93c/0xa90

Nov 27 21:14:41 liXXX-XXX kernel: [] ? capfilelock+0x0/0x10

Nov 27 21:14:41 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:14:41 liXXX-XXX kernel: [] ? capfilelock+0x0/0x10

Nov 27 21:14:41 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:14:41 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:14:41 liXXX-XXX kernel: [] ? capfilelock+0x0/0x10

Nov 27 21:14:41 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:14:41 liXXX-XXX kernel: Mem-Info:

Nov 27 21:14:41 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:14:41 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:14:41 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:14:42 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:14:42 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:14:42 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:14:42 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 169

Nov 27 21:14:42 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 153

Nov 27 21:14:42 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 97

Nov 27 21:14:42 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 158

Nov 27 21:14:42 liXXX-XXX kernel: activeanon:52288 inactiveanon:52335 isolated_anon:607

Nov 27 21:14:42 liXXX-XXX kernel: activefile:2553 inactivefile:959 isolated_file:1

Nov 27 21:14:42 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:0 unstable:0

Nov 27 21:14:43 liXXX-XXX kernel: free:1165 slabreclaimable:2255 slabunreclaimable:3630

Nov 27 21:14:43 liXXX-XXX kernel: mapped:4727 shmem:3919 pagetables:7329 bounce:0

Nov 27 21:14:43 liXXX-XXX kernel: DMA free:1988kB min:84kB low:104kB high:124kB activeanon:6096kB inactiveanon:6288kB activefile:64kB inactivefile:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slabreclaimable:100kB slabunreclaimable:92kB kernelstack:160kB pagetables:148kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:14:43 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:14:43 liXXX-XXX kernel: DMA32 free:2672kB min:2784kB low:3480kB high:4176kB activeanon:203056kB inactiveanon:203052kB activefile:10148kB inactivefile:3768kB unevictable:0kB isolated(anon):2428kB isolated(file):4kB present:500960kB mlocked:0kB dirty:0kB writeback:0kB mapped:18896kB shmem:15676kB slabreclaimable:8920kB slabunreclaimable:14428kB kernelstack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:3200 allunreclaimable? no

Nov 27 21:14:43 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:14:43 liXXX-XXX kernel: DMA: 14kB 08kB 416kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 1988kB

Nov 27 21:14:43 liXXX-XXX kernel: DMA32: 6684kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2672kB

Nov 27 21:14:43 liXXX-XXX kernel: 9437 total pagecache pages

Nov 27 21:14:43 liXXX-XXX kernel: 2006 pages in swap cache

Nov 27 21:14:43 liXXX-XXX kernel: Swap cache stats: add 18105, delete 16099, find 96926/97556

Nov 27 21:14:43 liXXX-XXX kernel: Free swap = 212148kB

Nov 27 21:14:43 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:14:43 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:14:43 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:14:43 liXXX-XXX kernel: 148048 pages shared

Nov 27 21:14:43 liXXX-XXX kernel: 117864 pages non-shared

Nov 27 21:14:43 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20

Nov 27 21:14:43 liXXX-XXX kernel: Pid: 20125, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:14:43 liXXX-XXX kernel: Call Trace:

Nov 27 21:14:43 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x584/0x670

Nov 27 21:14:43 liXXX-XXX kernel: [] ? cacheallocrefill+0x2e3/0x5c0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? kmemcachealloc+0xa7/0xb0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? skprotalloc+0x42/0x130

Nov 27 21:14:43 liXXX-XXX kernel: [] ? sk_clone+0x14/0x290

Nov 27 21:14:43 liXXX-XXX kernel: [] ? inetcskclone+0x10/0xb0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpcreateopenreq_child+0x21/0x470

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpv4synrecvsock+0x53/0x1d0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpv6synrecvsock+0x46d/0x580

Nov 27 21:14:43 liXXX-XXX kernel: [] ? rawspin_lock+0x5/0x10

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpcheckreq+0x31e/0x4a0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpv4do_rcv+0x1c7/0x2a0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? localbhenable+0x22/0xb0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? tcpv4rcv+0x73b/0x8a0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? iplocaldeliver_finish+0xcb/0x200

Nov 27 21:14:43 liXXX-XXX kernel: [] ? iprcvfinish+0x191/0x3b0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? ip_rcv+0x256/0x2e0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? netifreceiveskb+0x78/0x90

Nov 27 21:14:43 liXXX-XXX kernel: [] ? xennet_poll+0x59d/0xe20

Nov 27 21:14:43 liXXX-XXX kernel: [] ? runtimersoftirq+0x1b9/0x260

Nov 27 21:14:43 liXXX-XXX kernel: [] ? netrxaction+0x97/0x1b0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? _dosoftirq+0xae/0x140

Nov 27 21:14:43 liXXX-XXX kernel: [] ? unmask_evtchn+0x1a/0xf0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? call_softirq+0x1c/0x30

Nov 27 21:14:43 liXXX-XXX kernel: [] ? do_softirq+0x65/0xa0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? irq_exit+0x75/0x90

Nov 27 21:14:43 liXXX-XXX kernel: [] ? xenevtchndo_upcall+0x13e/0x200

Nov 27 21:14:43 liXXX-XXX kernel: [] ? xendohypervisor_callback+0x1e/0x30

Nov 27 21:14:43 liXXX-XXX kernel: [] ? hypercall_page+0x40a/0x1010

Nov 27 21:14:43 liXXX-XXX kernel: [] ? hypercall_page+0x40a/0x1010

Nov 27 21:14:43 liXXX-XXX kernel: [] ? notifyremotevia_irq+0x29/0x70

Nov 27 21:14:43 liXXX-XXX kernel: [] ? xenspinunlock_slow+0x32/0x70

Nov 27 21:14:43 liXXX-XXX kernel: [] ? page_referenced+0x222/0x230

Nov 27 21:14:43 liXXX-XXX kernel: [] ? shrinkpagelist+0x197/0x750

Nov 27 21:14:43 liXXX-XXX kernel: [] ? shrinkactivelist+0x25a/0x2d0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? determinedirtyablememory+0xd/0x20

Nov 27 21:14:43 liXXX-XXX kernel: [] ? isolatelrupages+0x8c/0x220

Nov 27 21:14:43 liXXX-XXX kernel: [] ? shrink_zone+0x64b/0xc80

Nov 27 21:14:43 liXXX-XXX kernel: [] ? trytofree_pages+0x1dd/0x3a0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x3b5/0x670

Nov 27 21:14:43 liXXX-XXX kernel: [] ? _dofault+0x3da/0x4e0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? handlemmfault+0x93c/0xa90

Nov 27 21:14:43 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? _switchto+0x26d/0x350

Nov 27 21:14:43 liXXX-XXX kernel: [] ? setnextentity+0x35/0x70

Nov 27 21:14:43 liXXX-XXX kernel: [] ? finishtaskswitch+0x52/0xd0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? pvclockclocksourceread+0x53/0xd0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? schedule+0x3a4/0x6e0

Nov 27 21:14:43 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:14:43 liXXX-XXX kernel: Mem-Info:

Nov 27 21:14:43 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:14:43 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:14:43 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:14:43 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:14:43 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:14:43 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:14:43 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 116

Nov 27 21:14:43 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 132

Nov 27 21:14:43 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 65

Nov 27 21:14:43 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 132

Nov 27 21:14:43 liXXX-XXX kernel: activeanon:52284 inactiveanon:52222 isolated_anon:855

Nov 27 21:14:43 liXXX-XXX kernel: activefile:2521 inactivefile:946 isolated_file:1

Nov 27 21:14:43 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:20 unstable:0

Nov 27 21:14:43 liXXX-XXX kernel: free:1196 slabreclaimable:2255 slabunreclaimable:3630

Nov 27 21:14:43 liXXX-XXX kernel: mapped:4699 shmem:3919 pagetables:7329 bounce:0

Nov 27 21:14:43 liXXX-XXX kernel: DMA free:1988kB min:84kB low:104kB high:124kB activeanon:6096kB inactiveanon:6288kB activefile:64kB inactivefile:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slabreclaimable:100kB slabunreclaimable:92kB kernelstack:160kB pagetables:148kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:14:44 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:14:45 liXXX-XXX kernel: DMA32 free:2796kB min:2784kB low:3480kB high:4176kB activeanon:203040kB inactiveanon:202600kB activefile:10020kB inactivefile:3716kB unevictable:0kB isolated(anon):3420kB isolated(file):4kB present:500960kB mlocked:0kB dirty:0kB writeback:80kB mapped:18784kB shmem:15676kB slabreclaimable:8920kB slabunreclaimable:14428kB kernelstack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:1782 allunreclaimable? no

Nov 27 21:14:45 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:14:45 liXXX-XXX kernel: DMA: 14kB 08kB 416kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 1988kB

Nov 27 21:14:45 liXXX-XXX kernel: DMA32: 6894kB 68kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2804kB

Nov 27 21:14:45 liXXX-XXX kernel: 9381 total pagecache pages

Nov 27 21:14:45 liXXX-XXX kernel: 2036 pages in swap cache

Nov 27 21:14:45 liXXX-XXX kernel: Swap cache stats: add 18158, delete 16122, find 96946/97581

Nov 27 21:14:45 liXXX-XXX kernel: Free swap = 212092kB

Nov 27 21:14:45 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:14:45 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:14:45 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:14:46 liXXX-XXX kernel: 147885 pages shared

Nov 27 21:14:46 liXXX-XXX kernel: 117974 pages non-shared

Nov 27 21:14:46 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20

Nov 27 21:14:46 liXXX-XXX kernel: Pid: 20096, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:14:46 liXXX-XXX kernel: Call Trace:

Nov 27 21:14:47 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x584/0x670

Nov 27 21:14:47 liXXX-XXX kernel: [] ? cacheallocrefill+0x2e3/0x5c0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? kmemcachealloc+0xa7/0xb0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? skprotalloc+0x42/0x130

Nov 27 21:14:47 liXXX-XXX kernel: [] ? sk_clone+0x14/0x290

Nov 27 21:14:47 liXXX-XXX kernel: [] ? inetcskclone+0x10/0xb0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpcreateopenreq_child+0x21/0x470

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpv4synrecvsock+0x53/0x1d0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpv6synrecvsock+0x46d/0x580

Nov 27 21:14:47 liXXX-XXX kernel: [] ? rawspin_lock+0x5/0x10

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpcheckreq+0x31e/0x4a0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpv4do_rcv+0x1c7/0x2a0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? localbhenable+0x22/0xb0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? tcpv4rcv+0x73b/0x8a0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? iplocaldeliver_finish+0xcb/0x200

Nov 27 21:14:47 liXXX-XXX kernel: [] ? iprcvfinish+0x191/0x3b0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? ip_rcv+0x256/0x2e0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? netifreceiveskb+0x78/0x90

Nov 27 21:14:47 liXXX-XXX kernel: [] ? xennet_poll+0x59d/0xe20

Nov 27 21:14:47 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:14:47 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:14:47 liXXX-XXX kernel: [] ? netrxaction+0x97/0x1b0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? _dosoftirq+0xae/0x140

Nov 27 21:14:47 liXXX-XXX kernel: [] ? unmask_evtchn+0x1a/0xf0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? call_softirq+0x1c/0x30

Nov 27 21:14:47 liXXX-XXX kernel: [] ? do_softirq+0x65/0xa0

Nov 27 21:14:47 liXXX-XXX kernel: [] ? irq_exit+0x75/0x90

Nov 27 21:14:47 liXXX-XXX kernel: [] ? xenevtchndo_upcall+0x13e/0x200

Nov 27 21:14:47 liXXX-XXX kernel: [] ? xendohypervisor_callback+0x1e/0x30

Nov 27 21:14:47 liXXX-XXX kernel: Mem-Info:

Nov 27 21:14:47 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:14:47 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:14:47 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:14:47 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:14:48 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:14:48 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:14:48 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 117

Nov 27 21:14:48 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 91

Nov 27 21:14:48 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 65

Nov 27 21:14:48 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 132

Nov 27 21:14:48 liXXX-XXX kernel: activeanon:52289 inactiveanon:52253 isolated_anon:855

Nov 27 21:14:48 liXXX-XXX kernel: activefile:2521 inactivefile:946 isolated_file:1

Nov 27 21:14:48 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:20 unstable:0

Nov 27 21:14:48 liXXX-XXX kernel: free:1196 slabreclaimable:2255 slabunreclaimable:3630

Nov 27 21:14:48 liXXX-XXX kernel: mapped:4699 shmem:3919 pagetables:7329 bounce:0

Nov 27 21:14:48 liXXX-XXX kernel: DMA free:2072kB min:84kB low:104kB high:124kB activeanon:6096kB inactiveanon:6208kB activefile:64kB inactivefile:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slabreclaimable:100kB slabunreclaimable:92kB kernelstack:160kB pagetables:148kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:14:48 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:14:48 liXXX-XXX kernel: DMA32 free:4788kB min:2784kB low:3480kB high:4176kB activeanon:203020kB inactiveanon:203024kB activefile:9668kB inactivefile:3644kB unevictable:0kB isolated(anon):1284kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:3552kB mapped:18224kB shmem:15452kB slabreclaimable:8844kB slabunreclaimable:14428kB kernelstack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:2149 allunreclaimable? no

Nov 27 21:14:48 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:14:48 liXXX-XXX kernel: DMA: 264kB 18kB 416kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2096kB

Nov 27 21:14:48 liXXX-XXX kernel: DMA32: 10194kB 818kB 416kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 4788kB

Nov 27 21:14:48 liXXX-XXX kernel: 10193 total pagecache pages

Nov 27 21:14:48 liXXX-XXX kernel: 2966 pages in swap cache

Nov 27 21:14:48 liXXX-XXX kernel: Swap cache stats: add 19097, delete 16131, find 96955/97594

Nov 27 21:14:48 liXXX-XXX kernel: Free swap = 208500kB

Nov 27 21:14:48 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:14:48 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:14:48 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:14:48 liXXX-XXX kernel: 147376 pages shared

Nov 27 21:14:48 liXXX-XXX kernel: 117343 pages non-shared

Nov 27 21:14:57 liXXX-XXX kernel: mysqld invoked oom-killer: gfpmask=0x4d0, order=2, oomadj=0

Nov 27 21:14:58 liXXX-XXX kernel: Pid: 20264, comm: mysqld Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:14:58 liXXX-XXX kernel: Call Trace:

Nov 27 21:14:58 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:14:58 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:14:58 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:14:58 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:14:58 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:14:58 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:14:58 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:14:58 liXXX-XXX kernel: [] ? cacheallocrefill+0x2e3/0x5c0

Nov 27 21:14:58 liXXX-XXX kernel: [] ? __kmalloc+0x10b/0x110

Nov 27 21:14:58 liXXX-XXX kernel: [] ? _allocskb+0x71/0x160

Nov 27 21:15:02 liXXX-XXX kernel: [] ? sockallocsend_pskb+0x1e7/0x300

Nov 27 21:15:03 liXXX-XXX kernel: [] ? unixstreamsendmsg+0x2b9/0x3d0

Nov 27 21:15:10 liXXX-XXX kernel: [] ? sockaiowrite+0x186/0x1a0

Nov 27 21:15:15 liXXX-XXX kernel: [] ? dosyncwrite+0xc0/0x100

Nov 27 21:15:22 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:23 liXXX-XXX kernel: [] ? pvclockclocksourceread+0x53/0xd0

Nov 27 21:15:23 liXXX-XXX kernel: [] ? vfs_write+0x18c/0x1a0

Nov 27 21:15:23 liXXX-XXX kernel: [] ? sys_write+0x53/0xa0

Nov 27 21:15:23 liXXX-XXX kernel: [] ? systemcallfastpath+0x16/0x1b

Nov 27 21:15:23 liXXX-XXX kernel: Mem-Info:

Nov 27 21:15:23 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:23 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:23 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 0

Nov 27 21:15:23 liXXX-XXX kernel: activeanon:48131 inactiveanon:48213 isolated_anon:168

Nov 27 21:15:23 liXXX-XXX kernel: activefile:202 inactivefile:540 isolated_file:25

Nov 27 21:15:23 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:0 unstable:0

Nov 27 21:15:23 liXXX-XXX kernel: free:1619 slabreclaimable:1643 slabunreclaimable:5785

Nov 27 21:15:23 liXXX-XXX kernel: mapped:2608 shmem:2532 pagetables:16779 bounce:0

Nov 27 21:15:23 liXXX-XXX kernel: DMA free:2008kB min:84kB low:104kB high:124kB activeanon:5732kB inactiveanon:5908kB activefile:20kB inactivefile:112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:76kB shmem:20kB slabreclaimable:68kB slabunreclaimable:300kB kernelstack:176kB pagetables:604kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:23 liXXX-XXX kernel: DMA32 free:4468kB min:2784kB low:3480kB high:4176kB activeanon:186792kB inactiveanon:186944kB activefile:788kB inactivefile:2048kB unevictable:0kB isolated(anon):672kB isolated(file):100kB present:500960kB mlocked:0kB dirty:0kB writeback:0kB mapped:10356kB shmem:10108kB slabreclaimable:6504kB slabunreclaimable:22840kB kernelstack:2040kB pagetables:66512kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:23 liXXX-XXX kernel: DMA: 24kB 48kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2008kB

Nov 27 21:15:23 liXXX-XXX kernel: DMA32: 8474kB 1198kB 616kB 132kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 4468kB

Nov 27 21:15:23 liXXX-XXX kernel: 12398 total pagecache pages

Nov 27 21:15:23 liXXX-XXX kernel: 9101 pages in swap cache

Nov 27 21:15:23 liXXX-XXX kernel: Swap cache stats: add 151688, delete 142587, find 138910/147611

Nov 27 21:15:23 liXXX-XXX kernel: Free swap = 0kB

Nov 27 21:15:23 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:15:23 liXXX-XXX kernel: ctiveanon:5780kB activefile:16kB inactivefile:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:52kB mapped:24kB shmem:20kB slabreclaimable:60kB slabunreclaimable:336kB kernelstack:176kB pagetables:588kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:160 all_unreclaimable? yes

Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:23 liXXX-XXX kernel: DMA32 free:2736kB min:2784kB low:3480kB high:4176kB activeanon:187752kB inactiveanon:187952kB activefile:1060kB inactivefile:2256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:188kB mapped:10668kB shmem:10084kB slabreclaimable:5736kB slabunreclaimable:22264kB kernelstack:2024kB pagetables:64824kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:5088 allunreclaimable? yes

Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:23 liXXX-XXX kernel: DMA: 04kB 108kB 116kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2016kB

Nov 27 21:15:23 liXXX-XXX kernel: DMA32: 6064kB 358kB 216kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2736kB

Nov 27 21:15:23 liXXX-XXX kernel: 11583 total pagecache pages

Nov 27 21:15:23 liXXX-XXX kernel: 8108 pages in swap cache

Nov 27 21:15:23 liXXX-XXX kernel: Swap cache stats: add 169548, delete 161440, find 142115/152306

Nov 27 21:15:24 liXXX-XXX kernel: Free swap = 12kB

Nov 27 21:15:25 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:15:35 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:15:41 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:15:41 liXXX-XXX kernel: 165215 pages shared

Nov 27 21:15:41 liXXX-XXX kernel: 120616 pages non-shared

Nov 27 21:15:41 liXXX-XXX kernel: Out of memory: kill process 20075 (httpd) score 85070 or a child

Nov 27 21:15:41 liXXX-XXX kernel: Killed process 20075 (httpd) vsz:340280kB, anon-rss:8176kB, file-rss:7444kB

Nov 27 21:15:41 liXXX-XXX kernel: httpd invoked oom-killer: gfpmask=0x280da, order=0, oomadj=0

Nov 27 21:15:41 liXXX-XXX kernel: Pid: 20228, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:15:41 liXXX-XXX kernel: Call Trace:

Nov 27 21:15:41 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:15:41 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:41 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:15:41 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:15:41 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:15:41 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:15:41 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:15:41 liXXX-XXX kernel: [] ? handlemmfault+0x93c/0xa90

Nov 27 21:15:41 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:42 liXXX-XXX kernel: [] ? xenmcflush+0x10c/0x1d0

Nov 27 21:15:47 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:15:49 liXXX-XXX kernel: [] ? xenwritemsr_safe+0x76/0xa0

Nov 27 21:15:50 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:51 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:53 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:15:53 liXXX-XXX kernel: [] ? filereadactor+0x6c/0x190

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:53 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:15:53 liXXX-XXX kernel: [] ? genericfileaio_read+0x4d7/0x6b0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? dosyncread+0xc0/0x100

Nov 27 21:15:53 liXXX-XXX kernel: [] ? mmap_region+0x225/0x530

Nov 27 21:15:53 liXXX-XXX kernel: [] ? vfs_read+0xc8/0x1a0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? sys_read+0x53/0xa0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? systemcallfastpath+0x16/0x1b

Nov 27 21:15:53 liXXX-XXX kernel: Mem-Info:

Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 25

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 30

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 2

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 77

Nov 27 21:15:53 liXXX-XXX kernel: activeanon:48649 inactiveanon:48757 isolated_anon:0

Nov 27 21:15:53 liXXX-XXX kernel: activefile:338 inactivefile:599 isolated_file:0

Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:86 unstable:0

Nov 27 21:15:53 liXXX-XXX kernel: free:1192 slabreclaimable:1428 slabunreclaimable:5645

Nov 27 21:15:53 liXXX-XXX kernel: mapped:2422 shmem:2519 pagetables:16218 bounce:0

Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2024kB min:84kB low:104kB high:124kB activeanon:5720kB inactiveanon:5828kB activefile:0kB inactivefile:104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:76kB shmem:16kB slabreclaimable:60kB slabunreclaimable:344kB kernelstack:176kB pagetables:588kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:7425 allunreclaimable? yes

Nov 27 21:15:53 liXXX-XXX kernel: l4>Mem-Info:

Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 40

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 18

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 90

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 178

Nov 27 21:15:53 liXXX-XXX kernel: activeanon:49216 inactiveanon:49275 isolated_anon:0

Nov 27 21:15:53 liXXX-XXX kernel: activefile:397 inactivefile:585 isolated_file:0

Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:123 unstable:0

Nov 27 21:15:53 liXXX-XXX kernel: free:1203 slabreclaimable:1305 slabunreclaimable:5409

Nov 27 21:15:53 liXXX-XXX kernel: mapped:1876 shmem:2469 pagetables:15350 bounce:0

Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2064kB min:84kB low:104kB high:124kB activeanon:5696kB inactiveanon:5584kB activefile:148kB inactivefile:212kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:88kB mapped:36kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:184kB pagetables:580kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:53 liXXX-XXX kernel: DMA32 free:2748kB min:2784kB low:3480kB high:4176kB activeanon:191168kB inactiveanon:191516kB activefile:1440kB inactivefile:2128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:404kB mapped:7468kB shmem:9868kB slabreclaimable:5168kB slabunreclaimable:21324kB kernelstack:1768kB pagetables:60820kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:4814 allunreclaimable? yes

Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:53 liXXX-XXX kernel: DMA: 154kB 68kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2076kB

Nov 27 21:15:53 liXXX-XXX kernel: DMA32: 6874kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2748kB

Nov 27 21:15:53 liXXX-XXX kernel: 10788 total pagecache pages

Nov 27 21:15:53 liXXX-XXX kernel: 7347 pages in swap cache

Nov 27 21:15:53 liXXX-XXX kernel: Swap cache stats: add 203670, delete 196323, find 144761/157218

Nov 27 21:15:53 liXXX-XXX kernel: Free swap = 0kB

Nov 27 21:15:53 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:15:53 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:15:53 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:15:53 liXXX-XXX kernel: 115797 pages shared

Nov 27 21:15:53 liXXX-XXX kernel: 121701 pages non-shared

Nov 27 21:15:53 liXXX-XXX kernel: Out of memory: kill process 20140 (httpd) score 84271 or a child

Nov 27 21:15:53 liXXX-XXX kernel: Killed process 20140 (httpd) vsz:337084kB, anon-rss:3676kB, file-rss:4212kB

Nov 27 21:15:53 liXXX-XXX kernel: httpd invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0

Nov 27 21:15:53 liXXX-XXX kernel: Pid: 20233, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:15:53 liXXX-XXX kernel: Call Trace:

Nov 27 21:15:53 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:53 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:15:53 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:15:53 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:15:53 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:15:53 liXXX-XXX kernel: [] ? _dopagecachereadahead+0xdc/0x230

Nov 27 21:15:53 liXXX-XXX kernel: [] ? wakebitfunction+0x0/0x30

Nov 27 21:15:53 liXXX-XXX kernel: [] ? ra_submit+0x1c/0x30

Nov 27 21:15:53 liXXX-XXX kernel: [] ? filemap_fault+0x3db/0x3e0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xensetpte_at+0x6e/0x100

Nov 27 21:15:53 liXXX-XXX kernel: [] ? _dofault+0x55/0x4e0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? handlemmfault+0x234/0xa90

Nov 27 21:15:53 liXXX-XXX kernel: [] ? error_exit+0x2a/0x60

Nov 27 21:15:53 liXXX-XXX kernel: [] ? retintrestoreargs+0x5/0x6

Nov 27 21:15:53 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:53 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:53 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:53 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:15:53 liXXX-XXX kernel: Mem-Info:

Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 40

Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 18

Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 90

Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 178

Nov 27 21:15:53 liXXX-XXX kernel: activeanon:49166 inactiveanon:49332 isolated_anon:0

Nov 27 21:15:53 liXXX-XXX kernel: activefile:397 inactivefile:604 isolated_file:0

Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:123 unstable:0

Nov 27 21:15:53 liXXX-XXX kernel: free:1196 slabreclaimable:1305 slabunreclaimable:5409

Nov 27 21:15:53 liXXX-XXX kernel: mapped:1904 shmem:2469 pagetables:15350 bounce:0

Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2036kB min:84kB low:104kB high:124kB activeanon:5496kB inactiveanon:5812kB activefile:148kB inactivefile:288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:88kB mapped:36kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:184kB pagetables:580kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:693 allunreclaimable? yes

Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:53 liXXX-XXX kernel: DMA32 free:2748kB min:2784kB low:3480kB high:4176kB activeanon:191168kB inactiveanon:191516kB activefile:1440kB inactivefile:2240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:404kB mapped:7580kB shmem:9868kB slabreclaimable:5168kB slabunreclaimable:21324kB kernelstack:1768kB pagetables:60820kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:4814 allunreclaimable? yes

Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:53 liXXX-XXX kernel: DMA: 04kB 48kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2000kB

Nov 27 21:15:53 liXXX-XXX kernel: DMA32: 6874kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2748kB

Nov 27 21:15:53 liXXX-XXX kernel: 10864 total pagecache pages

Nov 27 21:15:53 liXXX-XXX kernel: 7334 pages in swap cache

Nov 27 21:15:53 liXXX-XXX kernel: Swap cache stats: add 203674, delete 196340, find 144768/157239

Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 52kB

Nov 27 21:15:54 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:15:54 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:15:54 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:15:54 liXXX-XXX kernel: 114524 pages shared

Nov 27 21:15:54 liXXX-XXX kernel: 121186 pages non-shared

Nov 27 21:15:54 liXXX-XXX kernel: Out of memory: kill process 20141 (httpd) score 84271 or a child

Nov 27 21:15:54 liXXX-XXX kernel: Killed process 20141 (httpd) vsz:337084kB, anon-rss:3904kB, file-rss:4420kB

Nov 27 21:15:54 liXXX-XXX kernel: mysqld invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0

Nov 27 21:15:54 liXXX-XXX kernel: Pid: 4346, comm: mysqld Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:15:54 liXXX-XXX kernel: Call Trace:

Nov 27 21:15:54 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:54 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:15:54 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:15:54 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _dopagecachereadahead+0xdc/0x230

Nov 27 21:15:54 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:54 liXXX-XXX kernel: [] ? ra_submit+0x1c/0x30

Nov 27 21:15:54 liXXX-XXX kernel: [] ? filemap_fault+0x3db/0x3e0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xensetpte_at+0x6e/0x100

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _dofault+0x55/0x4e0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:54 liXXX-XXX kernel: [] ? capsocketaccept+0x0/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? handlemmfault+0x234/0xa90

Nov 27 21:15:54 liXXX-XXX kernel: [] ? capsocketaccept+0x0/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:54 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? rawspin_lock+0x5/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? moveaddrto_user+0xa4/0xb0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? sys_getsockname+0xa5/0x100

Nov 27 21:15:54 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:15:54 liXXX-XXX kernel: Mem-Info:

Nov 27 21:15:54 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 31

Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177

Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23

Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 102

Nov 27 21:15:54 liXXX-XXX kernel: activeanon:49006 inactiveanon:49384 isolated_anon:0

Nov 27 21:15:54 liXXX-XXX kernel: activefile:383 inactivefile:1254 isolated_file:0

Nov 27 21:15:54 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:32 unstable:0

Nov 27 21:15:54 liXXX-XXX kernel: free:1176 slabreclaimable:1299 slabunreclaimable:5317

Nov 27 21:15:54 liXXX-XXX kernel: mapped:1975 shmem:2434 pagetables:14978 bounce:0

Nov 27 21:15:54 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB activeanon:4680kB inactiveanon:4964kB activefile:60kB inactivefile:1656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:104kB mapped:52kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:176kB pagetables:500kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:5600 allunreclaimable? yes

Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:54 liXXX-XXX kernel: DMA32 free:2672kB min:2784kB low:3480kB high:4176kB activeanon:191344kB inactiveanon:192572kB activefile:1472kB inactivefile:3360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slabreclaimable:5144kB slabunreclaimable:20956kB kernelstack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:32613 allunreclaimable? yes

Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:54 liXXX-XXX kernel: DMA: 64kB 18kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2000kB

Nov 27 21:15:54 liXXX-XXX kernel: DMA32: 6624kB 18kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2672kB

Nov 27 21:15:54 liXXX-XXX kernel: 14343 total pagecache pages

Nov 27 21:15:54 liXXX-XXX kernel: 10243 pages in swap cache

Nov 27 21:15:54 liXXX-XXX kernel: Swap cache stats: add 219708, delete 209465, find 146269/160080

Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 20kB

Nov 27 21:15:54 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:15:54 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:15:54 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:15:54 liXXX-XXX kernel: 109516 pages shared

Nov 27 21:15:54 liXXX-XXX kernel: 121651 pages non-shared

Nov 27 21:15:54 liXXX-XXX kernel: Out of memory: kill process 19955 (httpd) score 84242 or a child

Nov 27 21:15:54 liXXX-XXX kernel: Killed process 19955 (httpd) vsz:336968kB, anon-rss:3804kB, file-rss:6308kB

Nov 27 21:15:54 liXXX-XXX kernel: httpd invoked oom-killer: gfpmask=0x200da, order=0, oomadj=0

Nov 27 21:15:54 liXXX-XXX kernel: Pid: 20208, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:15:54 liXXX-XXX kernel: Call Trace:

Nov 27 21:15:54 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:54 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:15:54 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:15:54 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:15:54 liXXX-XXX kernel: [] ? ptepsetaccess_flags+0x51/0x80

Nov 27 21:15:54 liXXX-XXX kernel: [] ? readswapcache_async+0xfc/0x150

Nov 27 21:15:54 liXXX-XXX kernel: [] ? swapin_readahead+0x7f/0xb0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? handlemmfault+0x7ab/0xa90

Nov 27 21:15:54 liXXX-XXX kernel: [] ? capfilelock+0x0/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:15:54 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:15:54 liXXX-XXX kernel: [] ? xen_clts+0x6b/0x80

Nov 27 21:15:54 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:15:54 liXXX-XXX kernel: Mem-Info:

Nov 27 21:15:54 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:15:54 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 29

Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177

Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23

Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 102

Nov 27 21:15:54 liXXX-XXX kernel: activeanon:49006 inactiveanon:49468 isolated_anon:0

Nov 27 21:15:54 liXXX-XXX kernel: activefile:383 inactivefile:1314 isolated_file:0

Nov 27 21:15:54 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:2 unstable:0

Nov 27 21:15:54 liXXX-XXX kernel: free:1083 slabreclaimable:1299 slabunreclaimable:5317

Nov 27 21:15:54 liXXX-XXX kernel: mapped:2015 shmem:2434 pagetables:14978 bounce:0

Nov 27 21:15:54 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB activeanon:4680kB inactiveanon:4964kB activefile:60kB inactivefile:1896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:212kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:176kB pagetables:500kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:5600 allunreclaimable? yes

Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:15:54 liXXX-XXX kernel: DMA32 free:2300kB min:2784kB low:3480kB high:4176kB activeanon:191344kB inactiveanon:192908kB activefile:1472kB inactivefile:3360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slabreclaimable:5144kB slabunreclaimable:20956kB kernelstack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:32613 allunreclaimable? yes

Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:15:54 liXXX-XXX kernel: DMA: 64kB 18kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2000kB

Nov 27 21:15:54 liXXX-XXX kernel: DMA32: 5754kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2300kB

Nov 27 21:15:54 liXXX-XXX kernel: 14449 total pagecache pages

Nov 27 21:15:54 liXXX-XXX kernel: 10288 pages in swap cache

Nov 27 21:15:54 liXXX-XXX kernel: Swap cache stats: add 219803, delete 209515, find 146301/160136

Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 220kB

Nov 27 21:15:57 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:16:01 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:16:03 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:16:03 liXXX-XXX kernel: 109633 pages shared

Nov 27 21:16:06 liXXX-XXX kernel: 121706 pages non-shared

Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 19979 (httpd) score 84242 or a child

Nov 27 21:16:06 liXXX-XXX kernel: Killed process 19979 (httpd) vsz:336968kB, anon-rss:4212kB, file-rss:6444kB

Nov 27 21:16:06 liXXX-XXX kernel: httpd invoked oom-killer: gfpmask=0x200da, order=0, oomadj=0

Nov 27 21:16:06 liXXX-XXX kernel: Pid: 20208, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:16:06 liXXX-XXX kernel: Call Trace:

Nov 27 21:16:06 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:16:06 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:16:06 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:16:06 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:16:06 liXXX-XXX kernel: [] ? ptepsetaccess_flags+0x51/0x80

Nov 27 21:16:06 liXXX-XXX kernel: [] ? readswapcache_async+0xfc/0x150

Nov 27 21:16:06 liXXX-XXX kernel: [] ? swapin_readahead+0x7f/0xb0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? handlemmfault+0x7ab/0xa90

Nov 27 21:16:06 liXXX-XXX kernel: [] ? capfilelock+0x0/0x10

Nov 27 21:16:06 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:16:06 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:16:06 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xen_clts+0x6b/0x80

Nov 27 21:16:06 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:16:06 liXXX-XXX kernel: Mem-Info:

Nov 27 21:16:06 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 155

Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177

Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23

Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 70

Nov 27 21:16:06 liXXX-XXX kernel: activeanon:48978 inactiveanon:49384 isolated_anon:0

Nov 27 21:16:06 liXXX-XXX kernel: activefile:383 inactivefile:1342 isolated_file:0

Nov 27 21:16:06 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:2 unstable:0

Nov 27 21:16:06 liXXX-XXX kernel: free:1083 slabreclaimable:1299 slabunreclaimable:5317

Nov 27 21:16:06 liXXX-XXX kernel: mapped:2015 shmem:2434 pagetables:14978 bounce:0

Nov 27 21:16:06 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB activeanon:4680kB inactiveanon:4964kB activefile:60kB inactivefile:1896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:212kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:176kB pagetables:500kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:5600 allunreclaimable? yes

Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:16:06 liXXX-XXX kernel: DMA32 free:2300kB min:2784kB low:3480kB high:4176kB activeanon:191232kB inactiveanon:192460kB activefile:1472kB inactivefile:3472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slabreclaimable:5144kB slabunreclaimable:20956kB kernelstack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:32613 allunreclaimable? yes

Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:16:06 liXXX-XXX kernel: DMA: 64kB 18kB 316kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2000kB

Nov 27 21:16:06 liXXX-XXX kernel: DMA32: 6684kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2672kB

Nov 27 21:16:06 liXXX-XXX kernel: 14365 total pagecache pages

Nov 27 21:16:06 liXXX-XXX kernel: 10184 pages in swap cache

Nov 27 21:16:06 liXXX-XXX kernel: Swap cache stats: add 219803, delete 209619, find 146301/160136

Nov 27 21:16:06 liXXX-XXX kernel: Free swap = 1796kB

Nov 27 21:16:06 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:16:06 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:16:06 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:16:06 liXXX-XXX kernel: 107825 pages shared

Nov 27 21:16:06 liXXX-XXX kernel: 121158 pages non-shared

Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 20032 (httpd) score 84238 or a child

Nov 27 21:16:06 liXXX-XXX kernel: Killed process 20032 (httpd) vsz:336952kB, anon-rss:6360kB, file-rss:6444kB

Nov 27 21:16:06 liXXX-XXX kernel: gamserver invoked oom-killer: gfpmask=0x201da, order=0, oom_adj=0

Nov 27 21:16:06 liXXX-XXX kernel: Pid: 8449, comm: gamserver Not tainted 2.6.35.4-x8664-linode16 #1

Nov 27 21:16:06 liXXX-XXX kernel: Call Trace:

Nov 27 21:16:06 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:16:06 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:16:06 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:16:06 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _dopagecachereadahead+0xdc/0x230

Nov 27 21:16:06 liXXX-XXX kernel: [] ? ra_submit+0x1c/0x30

Nov 27 21:16:06 liXXX-XXX kernel: [] ? filemap_fault+0x3db/0x3e0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? _dofault+0x55/0x4e0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? error_exit+0x2a/0x60

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenhypervisorcallback+0x1b/0x20

Nov 27 21:16:06 liXXX-XXX kernel: [] ? handlemmfault+0x234/0xa90

Nov 27 21:16:06 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:16:06 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:16:06 liXXX-XXX kernel: [] ? capfilepermission+0x0/0x10

Nov 27 21:16:06 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? pvclockclocksourceread+0x53/0xd0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? ktimegetts+0x69/0xd0

Nov 27 21:16:06 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:16:06 liXXX-XXX kernel: Mem-Info:

Nov 27 21:16:06 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0

Nov 27 21:16:06 liXXX-XXX kernel: DMA32 per-cpu:

Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 45

Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 76

Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 95

Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 134

Nov 27 21:16:06 liXXX-XXX kernel: activeanon:48624 inactiveanon:49273 isolated_anon:798

Nov 27 21:16:06 liXXX-XXX kernel: activefile:311 inactivefile:1743 isolated_file:2

Nov 27 21:16:06 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:176 unstable:0

Nov 27 21:16:06 liXXX-XXX kernel: free:1183 slabreclaimable:1279 slabunreclaimable:5273

Nov 27 21:16:06 liXXX-XXX kernel: mapped:2132 shmem:2389 pagetables:14434 bounce:0

Nov 27 21:16:06 liXXX-XXX kernel: DMA free:2020kB min:84kB low:104kB high:124kB activeanon:4160kB inactiveanon:4392kB activefile:0kB inactivefile:48kB unevictable:0kB isolated(anon):3072kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:700kB mapped:8kB shmem:8kB slabreclaimable:52kB slabunreclaimable:312kB kernelstack:168kB pagetables:516kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:14 allunreclaimable? yes

Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489

Nov 27 21:16:06 liXXX-XXX kernel: DMA32 free:2712kB min:2784kB low:3480kB high:4176kB activeanon:190336kB inactiveanon:192700kB activefile:1244kB inactivefile:6924kB unevictable:0kB isolated(anon):120kB isolated(file):8kB present:500960kB mlocked:0kB dirty:0kB writeback:4kB mapped:8520kB shmem:9548kB slabreclaimable:5064kB slabunreclaimable:20780kB kernelstack:1704kB pagetables:57220kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:6178 allunreclaimable? yes

Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0

Nov 27 21:16:06 liXXX-XXX kernel: DMA: 14kB 28kB 516kB 032kB 064kB 1128kB 1256kB 1512kB 11024kB 02048kB 0*4096kB = 2020kB

Nov 27 21:16:06 liXXX-XXX kernel: DMA32: 6784kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2712kB

Nov 27 21:16:06 liXXX-XXX kernel: 19692 total pagecache pages

Nov 27 21:16:06 liXXX-XXX kernel: 15245 pages in swap cache

Nov 27 21:16:06 liXXX-XXX kernel: Swap cache stats: add 237334, delete 222089, find 148153/163593

Nov 27 21:16:06 liXXX-XXX kernel: Free swap = 0kB

Nov 27 21:16:06 liXXX-XXX kernel: Total swap = 262140kB

Nov 27 21:16:06 liXXX-XXX kernel: 131071 pages RAM

Nov 27 21:16:06 liXXX-XXX kernel: 4842 pages reserved

Nov 27 21:16:06 liXXX-XXX kernel: 105720 pages shared

Nov 27 21:16:06 liXXX-XXX kernel: 120659 pages non-shared

Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 20051 (httpd) score 84238 or a child

Nov 27 21:16:06 liXXX-XXX kernel: Killed process 20051 (httpd) vsz:336952kB, anon-rss:6376kB, file-rss:6412kB

Nov 27 21:16:52 liXXX-XXX kernel: httpd invoked oom-killer: gfpmask=0x280da, order=0, oomadj=0

Nov 27 21:16:52 liXXX-XXX kernel: Pid: 20199, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1

Nov 27 21:16:52 liXXX-XXX kernel: Call Trace:

Nov 27 21:16:52 liXXX-XXX kernel: [] ? dump_header+0x72/0x1b0

Nov 27 21:16:52 liXXX-XXX kernel: [] ? xenrestorefldirectend+0x0/0x1

Nov 27 21:16:52 liXXX-XXX kernel: [] ? rawspinunlockirqrestore+0x26/0x40

Nov 27 21:16:52 liXXX-XXX kernel: [] ? oomkillprocess+0x91/0x170

Nov 27 21:16:52 liXXX-XXX kernel: [] ? _outof_memory+0x11d/0x190

Nov 27 21:16:52 liXXX-XXX kernel: [] ? outofmemory+0x5e/0xe0

Nov 27 21:16:52 liXXX-XXX kernel: [] ? _allocpages_nodemask+0x653/0x670

Nov 27 21:16:52 liXXX-XXX kernel: [] ? handlemmfault+0x93c/0xa90

Nov 27 21:16:52 liXXX-XXX kernel: [] ? xenforceevtchn_callback+0x9/0x10

Nov 27 21:16:52 liXXX-XXX kernel: [] ? check_events+0x12/0x20

Nov 27 21:17:02 liXXX-XXX kernel: [] ? capfilefree_security+0x0/0x10

Nov 27 21:17:07 liXXX-XXX kernel: [] ? dopagefault+0x1af/0x3c0

Nov 27 21:17:07 liXXX-XXX kernel: [] ? vma_merge+0x1eb/0x2e0

Nov 27 21:17:11 liXXX-XXX kernel: [] ? do_brk+0x253/0x340

Nov 27 21:17:13 liXXX-XXX kernel: [] ? dput+0xbc/0x180

Nov 27 21:17:15 liXXX-XXX kernel: [] ? page_fault+0x25/0x30

Nov 27 21:17:15 liXXX-XXX kernel: Mem-Info:

Nov 27 21:17:15 liXXX-XXX kernel: DMA per-cpu:

Nov 27 21:17:15 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0

Nov 27 21:17:15 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0

Nov 27 21:17:15 liXX

You OOM'd, what's your apache max clients set to?

Odds are really good you don't have apache tuned properly for your Linode which let too much memory get used when you happened to have a spike in usage. There are a number of tuning posts here you should be able find with a search. Try "MaxClients" as a search term.

Most likely quick 'n dirty hammer - lower "MaxClients" as it's too high for your size Linode and application/load.

– David

For an even quicker and dirtier piece of advice, lower MaxClients to 8.

Unless your processes are really really heavy (say >60MB, in which case, maybe try 4 instead) this should be low enough to fit Apache with mod_php nicely within a Linode 512. And if you need more processes, you'd probably know about it.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Log in to Reply
Community Code of Conduct