Page Allocation Failure
It's always coming from nginx, from what I've read this either means you're out of memory (which I'm not) or it's a kernel bug. Googling around suggests setting vm.lowerzoneprotection to 100 or more using sysctl.
This doesn't seem to be an option I'm using kernel 2.6.32-linode23 and I get error: "vm.lowerzoneprotection" is an unknown key.
My questions are
1) Anyone else had this and got rid of it?
2) Is it a kernel bug (and anyone know if there's a fix coming for it)?
The syslog entry is this (it's long).
Apr 4 05:29:42 zeus kernel: nginx: page allocation failure. order:1, mode:0x20
Apr 4 05:29:42 zeus kernel: Pid: 14798, comm: nginx Not tainted 2.6.32-linode23 #1
Apr 4 05:29:42 zeus kernel: Call Trace:
Apr 4 05:29:42 zeus kernel: [<c017a84c>] ? __alloc_pages_nodemask+0x49c/0x580
Apr 4 05:29:42 zeus kernel: [<c019c998>] ? cache_alloc_refill+0x2b8/0x520
Apr 4 05:29:42 zeus kernel: [<c019cd91>] ? kmem_cache_alloc+0xa1/0xb0
Apr 4 05:29:42 zeus kernel: [<c04b8f11>] ? sk_prot_alloc+0x31/0x130
Apr 4 05:29:42 zeus kernel: [<c04b9095>] ? sk_clone+0x15/0x200
Apr 4 05:29:42 zeus kernel: [<c051c7a7>] ? inet_csk_clone+0x17/0xa0
Apr 4 05:29:42 zeus kernel: [<c05317f2>] ? tcp_create_openreq_child+0x12/0x380
Apr 4 05:29:42 zeus kernel: [<c05301f8>] ? tcp_v4_syn_recv_sock+0x48/0x1a0
Apr 4 05:29:42 zeus kernel: [<c0531668>] ? tcp_check_req+0x2c8/0x440
Apr 4 05:29:42 zeus kernel: [<c052f989>] ? tcp_v4_do_rcv+0x119/0x1f0
Apr 4 05:29:42 zeus kernel: [<c05300d9>] ? tcp_v4_rcv+0x679/0x750
Apr 4 05:29:42 zeus kernel: [<c05137af>] ? ip_local_deliver_finish+0x9f/0x1f0
Apr 4 05:29:42 zeus kernel: [<c0513710>] ? ip_local_deliver_finish+0x0/0x1f0
Apr 4 05:29:42 zeus kernel: [<c0513243>] ? ip_rcv_finish+0x153/0x350
Apr 4 05:29:42 zeus kernel: [<c05130f0>] ? ip_rcv_finish+0x0/0x350
Apr 4 05:29:42 zeus kernel: [<c04c3fb7>] ? netif_receive_skb+0x307/0x4d0
Apr 4 05:29:42 zeus kernel: [<c04c41e7>] ? process_backlog+0x67/0xa0
Apr 4 05:29:42 zeus kernel: [<c04c4807>] ? net_rx_action+0xa7/0x140
Apr 4 05:29:42 zeus kernel: [<c01349a3>] ? __do_softirq+0x93/0x120
Apr 4 05:29:42 zeus kernel: [<c0461183>] ? loopback_xmit+0x53/0x80
Apr 4 05:29:42 zeus kernel: [<c04c4cd6>] ? dev_hard_start_xmit+0x196/0x370
Apr 4 05:29:42 zeus kernel: [<c0134a6d>] ? do_softirq+0x3d/0x40
Apr 4 05:29:42 zeus kernel: [<c0134cd5>] ? local_bh_enable+0x75/0x90
Apr 4 05:29:42 zeus kernel: [<c04c50b3>] ? dev_queue_xmit+0xf3/0x500
Apr 4 05:29:42 zeus kernel: [<c04f7def>] ? nf_hook_slow+0x9f/0xe0
Apr 4 05:29:42 zeus kernel: [<c05185b0>] ? ip_finish_output+0x0/0x2a0
Apr 4 05:29:42 zeus kernel: [<c0518797>] ? ip_finish_output+0x1e7/0x2a0
Apr 4 05:29:42 zeus kernel: [<c0517835>] ? ip_local_out+0x15/0x20
Apr 4 05:29:42 zeus kernel: [<c0517ff6>] ? ip_queue_xmit+0x1a6/0x3a0
Apr 4 05:29:42 zeus kernel: [<c011a1f5>] ? pvclock_clocksource_read+0x55/0xc0
Apr 4 05:29:42 zeus kernel: [<c0134c76>] ? local_bh_enable+0x16/0x90
Apr 4 05:29:42 zeus kernel: [<c0558837>] ? ipt_do_table+0x277/0x5c0
Apr 4 05:29:42 zeus kernel: [<c01052b7>] ? xen_force_evtchn_callback+0x17/0x30
Apr 4 05:29:42 zeus kernel: [<c052a76c>] ? tcp_transmit_skb+0x44c/0x690
Apr 4 05:29:42 zeus kernel: [<c052c83c>] ? tcp_write_xmit+0x1ec/0x9c0
Apr 4 05:29:42 zeus kernel: [<c052d07e>] ? __tcp_push_pending_frames+0x2e/0x90
Apr 4 05:29:42 zeus kernel: [<c052073d>] ? tcp_sendmsg+0x6cd/0xa70
Apr 4 05:29:42 zeus kernel: [<c04b55fd>] ? sock_recvmsg+0x12d/0x150
Apr 4 05:29:42 zeus kernel: [<c019cc89>] ? __kmalloc+0x89/0xf0
Apr 4 05:29:42 zeus kernel: [<c04b4dc4>] ? sock_aio_write+0x134/0x150
Apr 4 05:29:42 zeus kernel: [<c04b4c90>] ? sock_aio_write+0x0/0x150
Apr 4 05:29:42 zeus kernel: [<c01a130e>] ? do_sync_readv_writev+0xce/0x110
Apr 4 05:29:42 zeus kernel: [<c0144090>] ? autoremove_wake_function+0x0/0x50
Apr 4 05:29:42 zeus kernel: [<c03ce0dc>] ? security_file_permission+0xc/0x10
Apr 4 05:29:42 zeus kernel: [<c01a15ea>] ? rw_verify_area+0x5a/0xd0
Apr 4 05:29:42 zeus kernel: [<c01a1aee>] ? do_readv_writev+0x9e/0x1b0
Apr 4 05:29:42 zeus kernel: [<c04b4c90>] ? sock_aio_write+0x0/0x150
Apr 4 05:29:42 zeus kernel: [<c04b50cf>] ? sys_getsockopt+0xaf/0xc0
Apr 4 05:29:42 zeus kernel: [<c01a1c43>] ? vfs_writev+0x43/0x60
Apr 4 05:29:42 zeus kernel: [<c01a1d61>] ? sys_writev+0x41/0xb0
Apr 4 05:29:42 zeus kernel: [<c0107de9>] ? syscall_call+0x7/0xb
Apr 4 05:29:42 zeus kernel: Mem-Info:
Apr 4 05:29:42 zeus kernel: DMA per-cpu:
Apr 4 05:29:42 zeus kernel: CPU 0: hi: 0, btch: 1 usd: 0
Apr 4 05:29:42 zeus kernel: CPU 1: hi: 0, btch: 1 usd: 0
Apr 4 05:29:42 zeus kernel: CPU 2: hi: 0, btch: 1 usd: 0
Apr 4 05:29:42 zeus kernel: CPU 3: hi: 0, btch: 1 usd: 0
Apr 4 05:29:42 zeus kernel: Normal per-cpu:
Apr 4 05:29:42 zeus kernel: CPU 0: hi: 90, btch: 15 usd: 69
Apr 4 05:29:42 zeus kernel: CPU 1: hi: 90, btch: 15 usd: 23
Apr 4 05:29:42 zeus kernel: CPU 2: hi: 90, btch: 15 usd: 12
Apr 4 05:29:42 zeus kernel: CPU 3: hi: 90, btch: 15 usd: 31
Apr 4 05:29:42 zeus kernel: active_anon:9684 inactive_anon:14841 isolated_anon:0
Apr 4 05:29:42 zeus kernel: active_file:27988 inactive_file:30248 isolated_file:33
Apr 4 05:29:42 zeus kernel: unevictable:0 dirty:12939 writeback:1465 unstable:0
Apr 4 05:29:42 zeus kernel: free:948 slab_reclaimable:1719 slab_unreclaimable:1877
Apr 4 05:29:42 zeus kernel: mapped:3245 shmem:5224 pagetables:338 bounce:0
Apr 4 05:29:42 zeus kernel: DMA free:1460kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:652kB active_file:3228kB inactive_file:1672kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:108kB writeback:0kB mapped:64kB shmem:88kB slab_reclaimable:112kB slab_unreclaimable:528kB kernel_stack:224kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 4 05:29:42 zeus kernel: lowmem_reserve[]: 0 341 341 341
Apr 4 05:29:42 zeus kernel: Normal free:2332kB min:2308kB low:2884kB high:3460kB active_anon:38736kB inactive_anon:58712kB active_file:108724kB inactive_file:119320kB unevictable:0kB isolated(anon):0kB isolated(file):132kB present:349504kB mlocked:0kB dirty:51648kB writeback:5860kB mapped:12916kB shmem:20808kB slab_reclaimable:6764kB slab_unreclaimable:6980kB kernel_stack:816kB pagetables:1348kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 4 05:29:42 zeus kernel: lowmem_reserve[]: 0 0 0 0
Apr 4 05:29:42 zeus kernel: DMA: 366*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1504kB
Apr 4 05:29:42 zeus kernel: Normal: 555*4kB 8*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2332kB
Apr 4 05:29:42 zeus kernel: 64414 total pagecache pages
Apr 4 05:29:42 zeus kernel: 932 pages in swap cache
Apr 4 05:29:42 zeus kernel: Swap cache stats: add 79805, delete 78873, find 11919400/11926566
Apr 4 05:29:42 zeus kernel: Free swap = 254900kB
Apr 4 05:29:42 zeus kernel: Total swap = 262136kB
Apr 4 05:29:42 zeus kernel: 92160 pages RAM
Apr 4 05:29:42 zeus kernel: 0 pages HighMem
Apr 4 05:29:42 zeus kernel: 2989 pages reserved
Apr 4 05:29:42 zeus kernel: 50729 pages shared
Apr 4 05:29:42 zeus kernel: 50426 pages non-shared</c0107de9></c01a1d61></c01a1c43></c04b50cf></c04b4c90></c01a1aee></c01a15ea></c03ce0dc></c0144090></c01a130e></c04b4c90></c04b4dc4></c019cc89></c04b55fd></c052073d></c052d07e></c052c83c></c052a76c></c01052b7></c0558837></c0134c76></c011a1f5></c0517ff6></c0517835></c0518797></c05185b0></c04f7def></c04c50b3></c0134cd5></c0134a6d></c04c4cd6></c0461183></c01349a3></c04c4807></c04c41e7></c04c3fb7></c05130f0></c0513243></c0513710></c05137af></c05300d9></c052f989></c0531668></c05301f8></c05317f2></c051c7a7></c04b9095></c04b8f11></c019cd91></c019c998></c017a84c>
4 Replies
@obs:
out of memory (which I'm not)
Wouldn't be so sure – but it's not your fault, as all memory isn't the same. I see tracebacks like this when the box is under heavy fire networking-wise, and the networking driver runs out of buffer space or something. It is, as you've correctly guessed, a memory allocation failure in the kernel (and quite difficult to reproduce consistently).
Try 2.6.33-linode24 and see if this goes away.
@obs:
Kernel 24 isn't listed as the latest stable paravirt kernel, are there any issues I should know about before rebooting? (I'm on ubuntu 9.10)
None that I'm aware of.