Kernel page allocation failure, 2.6.38-linode31 and squeeze

Question

Kernel page allocation failure, 2.6.38-linode31 and squeeze

general

Hello,

Ever since I upgraded my Linode to Debian squeeze, I have been seeing regular kernel "page allocation failure" messages such as the following:

kernel: swapper: page allocation failure. order:4, mode:0x20
kernel: Pid: 0, comm: swapper Not tainted 2.6.38-linode31 #1
kernel: Call Trace:
kernel:  [<c01845c0>] ? __alloc_pages_nodemask+0x4f0/0x670
kernel:  [<c01aa259>] ? cache_alloc_refill+0x2f9/0x520
kernel:  [<c01aa534>] ? __kmalloc+0xb4/0xe0
kernel:  [<c04f43be>] ? pskb_expand_head+0x12e/0x200
kernel:  [<c010630b>] ? xen_restore_fl_direct_end+0x0/0x1
kernel:  [<c01a9666>] ? kmem_cache_free+0x46/0x120
kernel:  [<c04f490d>] ? __pskb_pull_tail+0x4d/0x2b0
kernel:  [<c05ee17d>] ? packet_rcv_spkt+0xfd/0x140
kernel:  [<c04fca7a>] ? dev_hard_start_xmit+0x26a/0x520
kernel:  [<c0510792>] ? sch_direct_xmit+0xb2/0x170
kernel:  [<c0534cc4>] ? nf_iterate+0x74/0xa0
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c04fce51>] ? dev_queue_xmit+0x121/0x550
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c0559dd4>] ? ip_finish_output+0x134/0x300
kernel:  [<c055a04a>] ? ip_output+0xaa/0xe0
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c0558f68>] ? ip_local_out+0x18/0x20
kernel:  [<c0559667>] ? ip_queue_xmit+0x137/0x3a0
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c056c652>] ? tcp_transmit_skb+0x372/0x7e0
kernel:  [<c056ec18>] ? tcp_write_xmit+0x198/0x980
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c056f464>] ? __tcp_push_pending_frames+0x24/0x90
kernel:  [<c056a3f6>] ? tcp_rcv_established+0x146/0x840
kernel:  [<c0571e46>] ? tcp_v4_do_rcv+0xd6/0x230
kernel:  [<c01382c6>] ? local_bh_enable+0x16/0x80
kernel:  [<c057266c>] ? tcp_v4_rcv+0x6cc/0x7b0
kernel:  [<c0554e37>] ? ip_local_deliver_finish+0x97/0x220
kernel:  [<c0554da0>] ? ip_local_deliver_finish+0x0/0x220
kernel:  [<c05547a6>] ? ip_rcv_finish+0xf6/0x3c0
kernel:  [<c04facfd>] ? __netif_receive_skb+0x32d/0x510
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c04fbbb7>] ? netif_receive_skb+0x67/0x70
kernel:  [<c049ff05>] ? xennet_poll+0x7f5/0xc20
kernel:  [<c04fc2fa>] ? net_rx_action+0x9a/0x130
kernel:  [<c013810c>] ? __do_softirq+0x7c/0x130
kernel:  [<c0138090>] ? __do_softirq+0x0/0x130
kernel:  <irq>[<c0138005>] ? irq_exit+0x65/0x70
kernel:  [<c0439f2d>] ? xen_evtchn_do_upcall+0x1d/0x30
kernel:  [<c0109487>] ? xen_do_upcall+0x7/0xc
kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1010
kernel:  [<c0105b8f>] ? xen_safe_halt+0xf/0x20
kernel:  [<c010f66f>] ? default_idle+0x2f/0x60
kernel:  [<c0107ed2>] ? cpu_idle+0x42/0x70
kernel:  [<c07ca8ac>] ? start_kernel+0x2da/0x2dfApr 12 06:38:43 li140-180 kernel:  [<c07ca410>] ? unknown_bootoption+0x0/0x190
kernel:  [<c07cdaa5>] ? xen_start_kernel+0x530/0x538
kernel: Mem-Info:
kernel: DMA per-cpu:
kernel: CPU    0: hi:    0, btch:   1 usd:   0
kernel: CPU    1: hi:    0, btch:   1 usd:   0
kernel: CPU    2: hi:    0, btch:   1 usd:   0
kernel: CPU    3: hi:    0, btch:   1 usd:   0
kernel: Normal per-cpu:
kernel: CPU    0: hi:  186, btch:  31 usd:  42
kernel: CPU    1: hi:  186, btch:  31 usd: 171
kernel: CPU    2: hi:  186, btch:  31 usd: 134
kernel: CPU    3: hi:  186, btch:  31 usd: 184
kernel: active_anon:16222 inactive_anon:27566 isolated_anon:0
kernel:  active_file:27915 inactive_file:31023 isolated_file:0
kernel:  unevictable:0 dirty:33 writeback:0 unstable:0
kernel:  free:9224 slab_reclaimable:9493 slab_unreclaimable:2108
kernel:  mapped:7298 shmem:4430 pagetables:611 bounce:0
kernel: DMA free:2084kB min:84kB low:104kB high:124kB active_anon:120kB inactive_anon:316kB active_file:2692kB inactive_file:2804kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:580kB shmem:408kB slab_reclaimable:72kB slab_unreclaimable:32kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 500 500 500
kernel: Normal free:34812kB min:2816kB low:3520kB high:4224kB active_anon:64768kB inactive_anon:109948kB active_file:108968kB inactive_file:121288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:132kB writeback:0kB mapped:28612kB shmem:17312kB slab_reclaimable:37900kB slab_unreclaimable:8400kB kernel_stack:1144kB pagetables:2444kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 0
kernel: DMA: 241*4kB 62*8kB 5*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2084kB
kernel: Normal: 2285*4kB 2697*8kB 225*16kB 17*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34860kB
kernel: 78061 total pagecache pages
kernel: 14676 pages in swap cache
kernel: Swap cache stats: add 330524, delete 315848, find 3444837/3487305
kernel: Free swap  = 423812kB
kernel: Total swap = 524284kB
kernel: 133104 pages RAM
kernel: 0 pages HighMem
kernel: 5637 pages reserved
kernel: 67797 pages shared
kernel: 65096 pages non-shared</c07cdaa5></c07ca410></c07ca8ac></c0107ed2></c010f66f></c0105b8f></c01013a7></c0109487></c0439f2d></c0138005></irq></c0138090></c013810c></c04fc2fa></c049ff05></c04fbbb7></c0105b37></c04facfd></c05547a6></c0554da0></c0554e37></c057266c></c01382c6></c0571e46></c056a3f6></c056f464></c0105b37></c056ec18></c056c652></c0105b37></c0105b37></c0559667></c0558f68></c0559ca0></c055a04a></c0559dd4></c0559ca0></c04fce51></c0559ca0></c0534cc4></c0510792></c04fca7a></c05ee17d></c04f490d></c01a9666></c010630b></c04f43be></c01aa534></c01aa259></c01845c0>

As indicated, this is using Linux 2.6.38-linode31.

The involved process is always "swapper", "apache2" or "ssh" and the problem most frequently occurs during a nightly script that rsyncs to a volume mounted over sshfs+encfs (rsync does not report any error).

This is the memory information that may be relevant:

# free -lm
             total       used       free     shared    buffers     cached
Mem:           497        426         71          0         38        166
Low:           497        426         71
High:            0          0          0
-/+ buffers/cache:        221        276
Swap:          511         77        434
# cat /proc/sys/vm/lowmem_reserve_ratio 
256    32  32
# cat /proc/zoneinfo 
Node 0, zone      DMA
  pages free     688
        min      21
        low      26
        high     31
        scanned  0
        spanned  4080
        present  3952
    nr_free_pages 688
    nr_inactive_anon 48
    nr_active_anon 83
    nr_inactive_file 578
    nr_active_file 584
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 18
    nr_mapped    150
    nr_file_pages 1279
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 9
    nr_slab_unreclaimable 0
    nr_page_table_pages 0
    nr_kernel_stack 21
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     102
    nr_dirtied   5617
    nr_written   5608
    nr_anon_transparent_hugepages 0
        protection: (0, 500, 500, 500)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
  all_unreclaimable: 0
  start_pfn:         16
  inactive_ratio:    1
Node 0, zone   Normal
  pages free     17443
        min      704
        low      880
        high     1056
        scanned  0
        spanned  129024
        present  128016
    nr_free_pages 17443
    nr_inactive_anon 33819
    nr_active_anon 18066
    nr_inactive_file 18962
    nr_active_file 27900
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 42007
    nr_mapped    8023
    nr_file_pages 61636
    nr_dirty     30
    nr_writeback 0
    nr_slab_reclaimable 2336
    nr_slab_unreclaimable 2213
    nr_page_table_pages 919
    nr_kernel_stack 171
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 232406
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     4470
    nr_dirtied   3852185
    nr_written   3961663
    nr_anon_transparent_hugepages 0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 184
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 1
              count: 176
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 2
              count: 165
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 3
              count: 164
              high:  186
              batch: 31
  vm stats threshold: 18
  all_unreclaimable: 0
  start_pfn:         4096
  inactive_ratio:    1

Any ideas what this is about?

Thanks,

Bruno

18 Replies

forum:Christopher Fritz · Answer 1 · April 13, 2011, 10:06 a.m.

forum:Christopher Fritz 13 years, 7 months ago

I'm seeing the same.

I was getting server freeze-ups with Centos, so I backed up all my data onto my PC, installed Debian, and this time went with the debs for Apache, MySQL, and PHP.

On CentOS, I had compiled nginx, MySQL, and PHP. I thought the fastcgi PHP processes for nginx were suddenly causing the freeze-ups, so I switched to Apache, but still got the freeze-ups. I increased the swap file from 256 MB to 512 MB, but still "ran out of memory". That's when I decided to go from Centos 5 to Debian 6.

With Debian, I'm still seeing the freeze-ups, with the "swapper: page allocation failure" error. I don't have enough information to wonder if it might be a hardware/ram error. Since the Linux OS and all the software used were replaced, I figure the cause has to be either something related to the data I migrated to the new install, or a hardware issue.

The freeze-ups for me usually end with something killing the MySQL process, which sometimes alleviates the freeze-up and sometimes doesn't. Because of this, I've run MySQL's MyISAM repair tool on my databases to see if that helps. (I have a MyBB forum installed, which gets a fair amount of use daily.) I did this about ten hours ago, so I should know in a few days if it's resolved the issues or not. I typically see two to three server freeze-ups a day, sometimes one being within an hour of rebooting. (Rebooting seems to be the only temporarily recovery solution for me.)

I'll have to remember to respond again later whether I do or do not see more freeze-ups.

Edit: I had three more freeze-ups later the day I wrote this. Guess it wasn't a problem with the database files!

forum:raindog308 · Answer 2 · April 14, 2011, 2:27 a.m.

forum:raindog308 13 years, 7 months ago

I've had the same issue. I'm also running Debian 6.

I asked support and they recommended increasing the kernel param vm.minfreekbytes to 16384. It was about 2900 before that.

I highly recommend you open a ticket with support and send them that dmesg output - don't just paste that setting :-)

forum:Christopher Fritz · Answer 3 · April 14, 2011, 10:42 a.m.

forum:Christopher Fritz 13 years, 7 months ago

I'll try changing that setting, and I'll read up on it a bit so I can understand what's happening. After I put in the change, I'll be sure to observe any changes in the server behavior, and see if a ticket should be put in (as there were issues with "default" settings.)

Edit: I still had a freeze-up even with vm.minfreekbytes set higher. I'll try what melon wrote below after I get home from work, and will report on my results. If things still don't work out, it'll be ticket time.

forum:melon · Answer 4 · April 14, 2011, 2:49 p.m.

forum:melon 13 years, 7 months ago

~~@raindog308:~~

I've had the same issue. I'm also running Debian 6.

I asked support and they recommended increasing the kernel param vm.minfreekbytes to 16384. It was about 2900 before that.

I highly recommend you open a ticket with support and send them that dmesg output - don't just paste that setting :-)
I'm having the very same issue, running Ubuntu Lucid. I ran the same laps with support. After all they advised booting 2.6.32.16-linode28. I rebooted this morning with this older kernel and I am seeing no failures so far, but I'll wait 1-2 days before reporting back.

forum:DaveE · Answer 5 · April 17, 2011, 6:32 p.m.

forum:DaveE 13 years, 7 months ago

I was also having this issue with ubuntu 10.04, support suggested I change vm.minfreekbytes to a higher value, it was 2906. I think they had suggested 16384, that solved it, I dropped it down to 8192 and still not seeing any issue.

to check it: sysctl vm.minfreekbytes

change it: sysctl vm.minfreekbytes=16384

to keep the change through a reboot you'll need to edit /etc/sysctl.conf

forum:defraine · Answer 6 · April 21, 2011, 3:30 a.m.

forum:defraine 13 years, 7 months ago

The issue was also solved by increasing vm.minfreekbytes and rebooting. Support suggested:

> sysctl vm.minfreekbytes

If the output of the above command is lower than "16384", please raise it by issuing the following command:

sysctl vm.minfreekbytes=16384

You'll also want to add this line to your "/etc/sysctl.conf" file to make the change stick across reboots:

vm/minfreekbytes = 16384

I had vm.minfreekbytes = 2906 before.

forum:Christopher Fritz · Answer 7 · April 22, 2011, 12:58 p.m.

forum:Christopher Fritz 13 years, 7 months ago

I thought for certain I updated my /etc/sysctl.conf file, but was still getting freeze ups, and checking the file showed I didn't have the necessary line added. I've made certain I edited the file this time, so here's hoping the freeze-ups will be gone for me now!

forum:archon810 · Answer 8 · July 27, 2011, 3:08 a.m.

forum:archon810 13 years, 3 months ago

Same issue here. Will try the sysctl vm.minfreekbytes solution. Damn you, xen!

sleddog · Answer 9 · July 27, 2011, 7:47 p.m.

sleddog 13 years, 3 months ago

~~@defraine:~~

The issue was also solved by increasing vm.minfreekbytes and rebooting. Support suggested:

> sysctl vm.minfreekbytes

If the output of the above command is lower than "16384", please raise it by issuing the following command:

sysctl vm.minfreekbytes=16384

You'll also want to add this line to your "/etc/sysctl.conf" file to make the change stick across reboots:

vm/minfreekbytes = 16384

Is that bolded line a typo?

forum:db3l · Answer 10 · July 27, 2011, 9:32 p.m.

forum:db3l 13 years, 3 months ago

I've also noticed that recent Linode kernels (32-bit at least) have some difference. The prior latest paravirt (2.6.39-linode33), at least as of changes included in build #3, increased the default vm.freeminkbytes to 4096 (from ~2900 in the 2.6.38 kernels), while the current latest paravirt (2.6.39.1-linode34) seems to be back to an older 2906. I wonder if a local Linode patch wasn't migrated forward, or are even the recent issues only on Linodes running a 2.6.38 kernel?

– David

forum:hoopycat · Answer 11 · July 27, 2011, 10:31 p.m.

forum:hoopycat 13 years, 3 months ago

This is pretty much an obsolete thread… the underlying problem was fixed in mainline 2.6.39, at least for 32-bit. Just reboot with Latest 2.6 Paravirt and you (should) be OK.

If you're running 64-bit… well, if you've always wanted your name in the kernel changelog, this is your chance.

forum:db3l · Answer 12 · July 27, 2011, 10:45 p.m.

forum:db3l 13 years, 3 months ago

~~@hoopycat:~~

This is pretty much an obsolete thread… the underlying problem was fixed in mainline 2.6.39, at least for 32-bit. Just reboot with Latest 2.6 Paravirt and you (should) be OK.
Do you have a reference for this? I assume you mean the underlying problem that the increased setting was a workaround for, and I'm curious what that issue was.

Thanks.

– David

forum:hoopycat · Answer 13 · July 27, 2011, 11:20 p.m.

forum:hoopycat 13 years, 3 months ago

http://forum.linode.com/viewtopic.php?t=7451

http://forum.linode.com/viewtopic.php?t=7308

Along with various chatter on IRC, a couple bits on http://www.linode.com/kernels/, etc.

forum:db3l · Answer 14 · July 28, 2011, 12:32 a.m.

forum:db3l 13 years, 3 months ago

~~@hoopycat:~~

http://forum.linode.com/viewtopic.php?t=7451

http://forum.linode.com/viewtopic.php?t=7308

Along with various chatter on IRC, a couple bits on http://www.linode.com/kernels/, etc.
Thanks.

It was the kernel listing that made me believe it was the change noted with 2.6.39-linode33 #3 ("increased minfreekbytes ratio calculation") that was intended to address the problem, and I did see the default value increase with that kernel, and subsequent 2.6.39-linode33 builds. But that appears to have gone away in 2.6.39.1 with the default value going back to 2.6.38 levels.

The other threads do appear, judging from post dates, to indicate that 2.6.39.1 should also be ok, but I'm curious how that's true with the lower default value. I was wondering if perhaps there was an upstream change that might have obviated the need for a larger value, or could 2.6.39.1 have re-introduced the susceptibility and people just haven't rebooted into it as much yet to have it happen?

– David

forum:wvolz · Answer 15 · Aug. 2, 2011, 7:30 p.m.

I'm still getting these on 2.6.39.1-linode34.

Sysctl:

vm.min_free_kbytes = 2906

Info in /var/log/messages:

Aug  2 15:46:22 linode kernel: pop3-login: page allocation failure. order:4, mode:0x20
Aug  2 15:46:22 linode kernel: Pid: 22575, comm: pop3-login Not tainted 2.6.39.1-linode34 #1
Aug  2 15:46:22 linode kernel: Call Trace:
Aug  2 15:46:22 linode kernel: [<c0189a30>] ? __alloc_pages_nodemask+0x530/0x6f0
Aug  2 15:46:22 linode kernel: [<c01afb13>] ? T.819+0xb3/0x2e0
Aug  2 15:46:22 linode kernel: [<c01aff86>] ? cache_alloc_refill+0x246/0x290
Aug  2 15:46:22 linode kernel: [<c0139826>] ? local_bh_enable+0x16/0x80
Aug  2 15:46:22 linode kernel: [<c01b008d>] ? __kmalloc+0xbd/0xd0
Aug  2 15:46:22 linode kernel: [<c050f07e>] ? pskb_expand_head+0x12e/0x200
Aug  2 15:46:22 linode kernel: [<c050f5bd>] ? __pskb_pull_tail+0x4d/0x2b0
Aug  2 15:46:22 linode kernel: [<c05d9263>] ? ipv4_confirm+0xd3/0x180
Aug  2 15:46:22 linode kernel: [<c0517d6d>] ? dev_hard_start_xmit+0x1dd/0x3e0
Aug  2 15:46:22 linode kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Aug  2 15:46:22 linode kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Aug  2 15:46:22 linode kernel: [<c052bcc2>] ? sch_direct_xmit+0xb2/0x170
Aug  2 15:46:22 linode kernel: [<c0518069>] ? dev_queue_xmit+0xf9/0x320
Aug  2 15:46:22 linode kernel: [<c059aa3b>] ? ip_finish_output+0x13b/0x300
Aug  2 15:46:22 linode kernel: [<c059acaa>] ? ip_output+0xaa/0xe0
Aug  2 15:46:22 linode kernel: [<c0599e78>] ? ip_local_out+0x18/0x20
Aug  2 15:46:22 linode kernel: [<c059a257>] ? ip_queue_xmit+0x117/0x3d0
Aug  2 15:46:22 linode kernel: [<c068fd1f>] ? _raw_spin_lock_bh+0xf/0x20
Aug  2 15:46:22 linode kernel: [<c050a4e3>] ? release_sock+0x13/0xd0
Aug  2 15:46:22 linode kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Aug  2 15:46:22 linode kernel: [<c01062c4>] ? check_events+0x8/0xc
Aug  2 15:46:22 linode kernel: [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Aug  2 15:46:22 linode kernel: [<c01b004f>] ? __kmalloc+0x7f/0xd0
Aug  2 15:46:22 linode kernel: [<c05ad70f>] ? tcp_transmit_skb+0x37f/0x660
Aug  2 15:46:22 linode kernel: [<c05b0165>] ? tcp_write_xmit+0x1e5/0x4f0
Aug  2 15:46:22 linode kernel: [<c05b049d>] ? tcp_push_one+0x2d/0x40
Aug  2 15:46:22 linode kernel: [<c05a343b>] ? tcp_sendmsg+0x9cb/0xaa0
Aug  2 15:46:22 linode kernel: [<c01062c4>] ? check_events+0x8/0xc
Aug  2 15:46:22 linode kernel: [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Aug  2 15:46:22 linode kernel: [<c068fb71>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Aug  2 15:46:22 linode kernel: [<c05f7a85>] ? unix_stream_recvmsg+0x4c5/0x5c0
Aug  2 15:46:22 linode kernel: [<c05c1407>] ? inet_sendmsg+0x47/0xb0
Aug  2 15:46:22 linode kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Aug  2 15:46:22 linode kernel: [<c0506ed9>] ? sock_aio_write+0x129/0x180
Aug  2 15:46:22 linode kernel: [<c01b2975>] ? do_sync_write+0xb5/0xf0
Aug  2 15:46:22 linode kernel: [<c01b2b06>] ? rw_verify_area+0x66/0x120
Aug  2 15:46:22 linode kernel: [<c05081ec>] ? sys_setsockopt+0x6c/0xd0
Aug  2 15:46:22 linode kernel: [<c01b35c4>] ? vfs_write+0x154/0x170
Aug  2 15:46:22 linode kernel: [<c01b36b1>] ? sys_write+0x41/0x80
Aug  2 15:46:22 linode kernel: [<c068fe51>] ? syscall_call+0x7/0xb
Aug  2 15:46:22 linode kernel: Mem-Info:
Aug  2 15:46:22 linode kernel: DMA per-cpu:
Aug  2 15:46:22 linode kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug  2 15:46:22 linode kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug  2 15:46:22 linode kernel: CPU    2: hi:    0, btch:   1 usd:   0
Aug  2 15:46:22 linode kernel: CPU    3: hi:    0, btch:   1 usd:   0
Aug  2 15:46:22 linode kernel: Normal per-cpu:
Aug  2 15:46:22 linode kernel: CPU    0: hi:  186, btch:  31 usd:  81
Aug  2 15:46:22 linode kernel: CPU    1: hi:  186, btch:  31 usd: 144
Aug  2 15:46:22 linode kernel: CPU    2: hi:  186, btch:  31 usd: 183
Aug  2 15:46:22 linode kernel: CPU    3: hi:  186, btch:  31 usd: 115
Aug  2 15:46:22 linode kernel: active_anon:14006 inactive_anon:23440 isolated_anon:0
Aug  2 15:46:22 linode kernel: active_file:40133 inactive_file:16338 isolated_file:0
Aug  2 15:46:22 linode kernel: unevictable:0 dirty:15 writeback:0 unstable:0
Aug  2 15:46:22 linode kernel: free:22792 slab_reclaimable:3207 slab_unreclaimable:3175
Aug  2 15:46:22 linode kernel: mapped:5907 shmem:918 pagetables:730 bounce:0
Aug  2 15:46:22 linode kernel: DMA free:2056kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:2984kB inactive_file:1604kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  2 15:46:22 linode kernel: lowmem_reserve[]: 0 500 500 500
Aug  2 15:46:22 linode kernel: Normal free:89112kB min:2816kB low:3520kB high:4224kB active_anon:56024kB inactive_anon:93760kB active_file:157548kB inactive_file:63748kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:60kB writeback:0kB mapped:23628kB shmem:3672kB slab_reclaimable:12816kB slab_unreclaimable:12668kB kernel_stack:1400kB pagetables:2920kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  2 15:46:22 linode kernel: lowmem_reserve[]: 0 0 0 0
Aug  2 15:46:22 linode kernel: DMA: 2*4kB 2*8kB 5*16kB 9*32kB 10*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2056kB
Aug  2 15:46:22 linode kernel: Normal: 3118*4kB 9081*8kB 192*16kB 29*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 89120kB
Aug  2 15:46:22 linode kernel: 65672 total pagecache pages
Aug  2 15:46:22 linode kernel: 8293 pages in swap cache
Aug  2 15:46:22 linode kernel: Swap cache stats: add 956363, delete 948070, find 38950697/39101773
Aug  2 15:46:22 linode kernel: Free swap  = 444508kB
Aug  2 15:46:22 linode kernel: Total swap = 524284kB
Aug  2 15:46:22 linode kernel: 133104 pages RAM
Aug  2 15:46:22 linode kernel: 0 pages HighMem
Aug  2 15:46:22 linode kernel: 5748 pages reserved
Aug  2 15:46:22 linode kernel: 40759 pages shared
Aug  2 15:46:22 linode kernel: 82822 pages non-shared</c068fe51></c01b36b1></c01b35c4></c05081ec></c01b2b06></c01b2975></c0506ed9></c0105ae7></c05c1407></c05f7a85></c068fb71></c01062bb></c01062c4></c05a343b></c05b049d></c05b0165></c05ad70f></c01b004f></c01062bb></c01062c4></c0105ae7></c050a4e3></c068fd1f></c059a257></c0599e78></c059acaa></c059aa3b></c0518069></c052bcc2></c059a900></c059a900></c0517d6d></c05d9263></c050f5bd></c050f07e></c01b008d></c0139826></c01aff86></c01afb13></c0189a30>

forum:sweh · Answer 16 · Aug. 2, 2011, 7:38 p.m.

I just got a bunch of these yesterday; I was doing a recursive "lsattr" in order to ensure I had the backup settings correct, and then I got a few of those memory errors :-(

Linux 2.6.39.1-linode34 #1 SMP

Interesting it's kworker that's aborting this time…

Aug  1 11:41:30 linode kernel: kworker/0:2: page allocation failure. order:4, mode:0x20
Aug  1 11:41:30 linode kernel: Pid: 26003, comm: kworker/0:2 Not tainted 2.6.39.1-linode34 #1
Aug  1 11:41:30 linode kernel: Call Trace:
Aug  1 11:41:30 linode kernel:  [<c0189a30>] ? __alloc_pages_nodemask+0x530/0x6f0
Aug  1 11:41:30 linode kernel:  [<c015a740>] ? clockevents_program_event+0xa0/0x160
Aug  1 11:41:30 linode kernel:  [<c01afb13>] ? T.819+0xb3/0x2e0
Aug  1 11:41:30 linode kernel:  [<c01aff86>] ? cache_alloc_refill+0x246/0x290
Aug  1 11:41:30 linode kernel:  [<c0139826>] ? local_bh_enable+0x16/0x80
Aug  1 11:41:30 linode kernel:  [<c01b008d>] ? __kmalloc+0xbd/0xd0
Aug  1 11:41:30 linode kernel:  [<c050f07e>] ? pskb_expand_head+0x12e/0x200
Aug  1 11:41:30 linode kernel:  [<c050f5bd>] ? __pskb_pull_tail+0x4d/0x2b0
Aug  1 11:41:30 linode kernel:  [<c05d9263>] ? ipv4_confirm+0xd3/0x180
Aug  1 11:41:30 linode kernel:  [<c0517d6d>] ? dev_hard_start_xmit+0x1dd/0x3e0
Aug  1 11:41:30 linode kernel:  [<c059a900>] ? ip_finish_output2+0x260/0x260
Aug  1 11:41:30 linode kernel:  [<c059a900>] ? ip_finish_output2+0x260/0x260
Aug  1 11:41:30 linode kernel:  [<c052bcc2>] ? sch_direct_xmit+0xb2/0x170
Aug  1 11:41:30 linode kernel:  [<c0518069>] ? dev_queue_xmit+0xf9/0x320
Aug  1 11:41:30 linode kernel:  [<c059aa3b>] ? ip_finish_output+0x13b/0x300
Aug  1 11:41:30 linode kernel:  [<c059acaa>] ? ip_output+0xaa/0xe0
Aug  1 11:41:30 linode kernel:  [<c0599e78>] ? ip_local_out+0x18/0x20
Aug  1 11:41:30 linode kernel:  [<c059a257>] ? ip_queue_xmit+0x117/0x3d0
Aug  1 11:41:30 linode kernel:  [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Aug  1 11:41:30 linode kernel:  [<c01062c4>] ? check_events+0x8/0xc
Aug  1 11:41:30 linode kernel:  [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Aug  1 11:41:30 linode kernel:  [<c01b004f>] ? __kmalloc+0x7f/0xd0
Aug  1 11:41:30 linode kernel:  [<c05ad70f>] ? tcp_transmit_skb+0x37f/0x660
Aug  1 11:41:30 linode kernel:  [<c05b0165>] ? tcp_write_xmit+0x1e5/0x4f0
Aug  1 11:41:30 linode kernel:  [<c05b04d4>] ? __tcp_push_pending_frames+0x24/0x90
Aug  1 11:41:30 linode kernel:  [<c05ac4e2>] ? tcp_rcv_established+0x3d2/0x610
Aug  1 11:41:30 linode kernel:  [<c05b2fee>] ? tcp_v4_do_rcv+0xce/0x170
Aug  1 11:41:30 linode kernel:  [<c05b3749>] ? tcp_v4_rcv+0x6b9/0x7a0
Aug  1 11:41:30 linode kernel:  [<c0595887>] ? ip_local_deliver_finish+0x97/0x220
Aug  1 11:41:30 linode kernel:  [<c05957f0>] ? ip_rcv+0x320/0x320
Aug  1 11:41:30 linode kernel:  [<c059524b>] ? ip_rcv_finish+0xfb/0x380
Aug  1 11:41:30 linode kernel:  [<c0516ca9>] ? __netif_receive_skb+0x339/0x3d0
Aug  1 11:41:30 linode kernel:  [<c0516f47>] ? netif_receive_skb+0x67/0x70
Aug  1 11:41:30 linode kernel:  [<c04b84cc>] ? handle_incoming_queue+0x17c/0x250
Aug  1 11:41:30 linode kernel:  [<c04b87bc>] ? xennet_poll+0x21c/0x540
Aug  1 11:41:30 linode kernel:  [<c05176da>] ? net_rx_action+0xea/0x190
Aug  1 11:41:30 linode kernel:  [<c013956c>] ? __do_softirq+0x7c/0x110
Aug  1 11:41:30 linode kernel:  [<c01394f0>] ? __local_bh_enable+0x70/0x70
Aug  1 11:41:30 linode kernel:  <irq>[<c013944e>] ? irq_exit+0x6e/0x90
Aug  1 11:41:30 linode kernel:  [<c044d14d>] ? xen_evtchn_do_upcall+0x1d/0x30
Aug  1 11:41:30 linode kernel:  [<c0690c07>] ? xen_do_upcall+0x7/0xc
Aug  1 11:41:30 linode kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Aug  1 11:41:30 linode kernel:  [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Aug  1 11:41:30 linode kernel:  [<c01062c4>] ? check_events+0x8/0xc
Aug  1 11:41:30 linode kernel:  [<c0106283>] ? xen_irq_enable_direct_reloc+0x4/0x4
Aug  1 11:41:30 linode kernel:  [<c012f9f7>] ? finish_task_switch+0x47/0xb0
Aug  1 11:41:30 linode kernel:  [<c068de3c>] ? schedule+0x1ec/0x6d0
Aug  1 11:41:30 linode kernel:  [<c01062c4>] ? check_events+0x8/0xc
Aug  1 11:41:30 linode kernel:  [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Aug  1 11:41:30 linode kernel:  [<c068fb71>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Aug  1 11:41:30 linode kernel:  [<c068fb50>] ? _raw_spin_lock_irq+0x10/0x20
Aug  1 11:41:30 linode kernel:  [<c0148e8a>] ? process_one_work+0x12a/0x360
Aug  1 11:41:30 linode kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Aug  1 11:41:30 linode kernel:  [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Aug  1 11:41:30 linode kernel:  [<c045bc00>] ? tty_buffer_free_all+0x70/0x70
Aug  1 11:41:30 linode kernel:  [<c0149506>] ? worker_thread+0x1c6/0x3c0
Aug  1 11:41:30 linode kernel:  [<c0149340>] ? rescuer_thread+0x280/0x280
Aug  1 11:41:30 linode kernel:  [<c014c8f4>] ? kthread+0x74/0x80
Aug  1 11:41:30 linode kernel:  [<c014c880>] ? kthread_worker_fn+0x170/0x170
Aug  1 11:41:30 linode kernel:  [<c0690bb6>] ? kernel_thread_helper+0x6/0x10
Aug  1 11:41:30 linode kernel: Mem-Info:
Aug  1 11:41:30 linode kernel: DMA per-cpu:
Aug  1 11:41:30 linode kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug  1 11:41:30 linode kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug  1 11:41:30 linode kernel: CPU    2: hi:    0, btch:   1 usd:   0
Aug  1 11:41:30 linode kernel: CPU    3: hi:    0, btch:   1 usd:   0
Aug  1 11:41:30 linode kernel: Normal per-cpu:
Aug  1 11:41:30 linode kernel: CPU    0: hi:  186, btch:  31 usd: 156
Aug  1 11:41:30 linode kernel: CPU    1: hi:  186, btch:  31 usd: 171
Aug  1 11:41:30 linode kernel: CPU    2: hi:  186, btch:  31 usd: 180
Aug  1 11:41:30 linode kernel: CPU    3: hi:  186, btch:  31 usd: 161
Aug  1 11:41:30 linode kernel: active_anon:746 inactive_anon:2104 isolated_anon:0
Aug  1 11:41:30 linode kernel:  active_file:13769 inactive_file:13373 isolated_file:0
Aug  1 11:41:30 linode kernel:  unevictable:1136 dirty:631 writeback:0 unstable:0
Aug  1 11:41:30 linode kernel:  free:15438 slab_reclaimable:74596 slab_unreclaimable:2754
Aug  1 11:41:30 linode kernel:  mapped:2167 shmem:8 pagetables:284 bounce:0
Aug  1 11:41:30 linode kernel: DMA free:2060kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:1124kB inactive_file:2796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:648kB slab_unreclaimable:60kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  1 11:41:30 linode kernel: lowmem_reserve[]: 0 500 500 500
Aug  1 11:41:30 linode kernel: Normal free:59692kB min:2816kB low:3520kB high:4224kB active_anon:2984kB inactive_anon:8416kB active_file:53952kB inactive_file:50696kB unevictable:4544kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:4544kB dirty:2524kB writeback:0kB mapped:8668kB shmem:32kB slab_reclaimable:297736kB slab_unreclaimable:10956kB kernel_stack:872kB pagetables:1136kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug  1 11:41:30 linode kernel: lowmem_reserve[]: 0 0 0 0
Aug  1 11:41:30 linode kernel: DMA: 50*4kB 49*8kB 32*16kB 14*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2064kB
Aug  1 11:41:30 linode kernel: Normal: 9067*4kB 2296*8kB 294*16kB 9*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59692kB
Aug  1 11:41:30 linode kernel: 29005 total pagecache pages
Aug  1 11:41:30 linode kernel: 953 pages in swap cache
Aug  1 11:41:30 linode kernel: Swap cache stats: add 13730, delete 12777, find 142314/144061
Aug  1 11:41:30 linode kernel: Free swap  = 254888kB
Aug  1 11:41:30 linode kernel: Total swap = 263164kB
Aug  1 11:41:30 linode kernel: 133104 pages RAM
Aug  1 11:41:30 linode kernel: 0 pages HighMem
Aug  1 11:41:30 linode kernel: 5748 pages reserved
Aug  1 11:41:31 linode kernel: 36833 pages shared
Aug  1 11:41:31 linode kernel: 82780 pages non-shared</c0690bb6></c014c880></c014c8f4></c0149340></c0149506></c045bc00></c0105ae7></c0101227></c0148e8a></c068fb50></c068fb71></c01062bb></c01062c4></c068de3c></c012f9f7></c0106283></c01062c4></c0105ae7></c0101227></c0690c07></c044d14d></c013944e></irq></c01394f0></c013956c></c05176da></c04b87bc></c04b84cc></c0516f47></c0516ca9></c059524b></c05957f0></c0595887></c05b3749></c05b2fee></c05ac4e2></c05b04d4></c05b0165></c05ad70f></c01b004f></c01062bb></c01062c4></c0105ae7></c059a257></c0599e78></c059acaa></c059aa3b></c0518069></c052bcc2></c059a900></c059a900></c0517d6d></c05d9263></c050f5bd></c050f07e></c01b008d></c0139826></c01aff86></c01afb13></c015a740></c0189a30>

forum:ferodynamics · Answer 17 · Oct. 8, 2013, 1:59 p.m.

forum:ferodynamics 11 years, 1 month ago

Is there any harm in setting it higher than 16384 to 65536?

Right now I'm at vm.minfreekbytes = 3429

I found this on the Fedora website, which is what I'm using on the Linode that went down.

> Unable to Allocate Memory / page allocation failure

Heavily-loaded network servers may have trouble allocating memory even though there is no shortage. Try setting the sysctl vm.minfreekbytes to 65536 in order to keep additional memory free for allocation by network drivers.

Same error 16 days ago but that was "order:2" and today was "order:4" whatever that means. Looking at Linode graphs, just before the crash outgoing traffic was getting up over 5 Mbits/second (only around 3 Mbits/second in the morning) so I wonder if this "panic" was due to the server working harder. (Last night added more images to the website.)

Maybe this thread is old but it's nice to see other people were able to fix their "page allocation failure".

forum:theckman · Answer 18 · Oct. 8, 2013, 2:04 p.m.

forum:theckman 11 years, 1 month ago

If you are still on our 2.6.x tree of kernels you need to upgrade. I also believe these issues are fixed in newer kernels. In addition to that, the 2.6.x line have a local root-escalation bug that's a bit concerning.

I'd suggest selecting the Latest 32-bit (or 64-bit if your OS is 64-bit) kernel in the Configuration Profile editor in the Linode Manager and then reboot.

-Tim

Compute

Storage

Networking

Databases

Services

Developer Tools

Industries

Pricing

Community

Engage With Us

Kernel page allocation failure, 2.6.38-linode31 and squeeze

18 Replies

Reply

Tips: