Kernel Errors Present - Any ideas?

I am getting the error below from my log watch files… Any idea what they mean?

WARNING: Kernel Errors Present

[] ? error_code+0x66/0x6c …: 5 Time(s)

9 Time(s): 0 pages HighMem

9 Time(s): 0 pages in swap cache

1 Time(s): 18226 pages shared

1 Time(s): 19813 pages shared

1 Time(s): 19877 pages shared

1 Time(s): 20036 pages shared

1 Time(s): 20064 pages shared

1 Time(s): 20190 pages shared

1 Time(s): 20220 pages shared

1 Time(s): 20251 pages shared

1 Time(s): 20255 pages shared

9 Time(s): 2989 pages reserved

2 Time(s): 7829 total pagecache pages

1 Time(s): 7851 total pagecache pages

1 Time(s): 7905 total pagecache pages

1 Time(s): 7907 total pagecache pages

1 Time(s): 7944 total pagecache pages

1 Time(s): 8062 total pagecache pages

1 Time(s): 8144 total pagecache pages

1 Time(s): 83358 pages non-shared

1 Time(s): 83960 pages non-shared

1 Time(s): 84086 pages non-shared

1 Time(s): 8485 total pagecache pages

1 Time(s): 84959 pages non-shared

1 Time(s): 85415 pages non-shared

1 Time(s): 85642 pages non-shared

1 Time(s): 85707 pages non-shared

1 Time(s): 85739 pages non-shared

1 Time(s): 85775 pages non-shared

9 Time(s): 92160 pages RAM

9 Time(s): CPU 0: hi: 0, btch: 1 usd: 0

3 Time(s): CPU 0: hi: 90, btch: 15 usd: 0

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 14

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 17

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 33

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 51

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 79

1 Time(s): CPU 0: hi: 90, btch: 15 usd: 86

9 Time(s): CPU 1: hi: 0, btch: 1 usd: 0

6 Time(s): CPU 1: hi: 90, btch: 15 usd: 0

1 Time(s): CPU 1: hi: 90, btch: 15 usd: 65

2 Time(s): CPU 1: hi: 90, btch: 15 usd: 83

9 Time(s): CPU 2: hi: 0, btch: 1 usd: 0

8 Time(s): CPU 2: hi: 90, btch: 15 usd: 0

1 Time(s): CPU 2: hi: 90, btch: 15 usd: 69

9 Time(s): CPU 3: hi: 0, btch: 1 usd: 0

8 Time(s): CPU 3: hi: 90, btch: 15 usd: 0

1 Time(s): CPU 3: hi: 90, btch: 15 usd: 58

9 Time(s): Call Trace:

1 Time(s): DMA free:1452kB min:104kB low:128kB high:156kB activeanon:3216kB inactiveanon:3088kB activefile:40kB inactivefile:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slabreclaimable:64kB slabunreclaimable:12kB kernelstack:32kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): DMA free:1456kB min:104kB low:128kB high:156kB activeanon:3084kB inactiveanon:3316kB activefile:4kB inactivefile:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slabreclaimable:28kB slabunreclaimable:12kB kernelstack:8kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): DMA free:1456kB min:104kB low:128kB high:156kB activeanon:3164kB inactiveanon:3088kB activefile:8kB inactivefile:116kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:0kB slabreclaimable:44kB slabunreclaimable:16kB kernelstack:24kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

2 Time(s): DMA free:1464kB min:104kB low:128kB high:156kB activeanon:3140kB inactiveanon:3172kB activefile:24kB inactivefile:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slabreclaimable:28kB slabunreclaimable:12kB kernelstack:32kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

1 Time(s): DMA free:1468kB min:104kB low:128kB high:156kB activeanon:2920kB inactiveanon:3240kB activefile:0kB inactivefile:200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:0kB slabreclaimable:32kB slabunreclaimable:24kB kernelstack:8kB pagetables:32kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

1 Time(s): DMA free:1508kB min:104kB low:128kB high:156kB activeanon:3012kB inactiveanon:3236kB activefile:4kB inactivefile:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slabreclaimable:32kB slabunreclaimable:12kB kernelstack:8kB pagetables:100kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

2 Time(s): DMA free:1644kB min:104kB low:128kB high:156kB activeanon:2972kB inactiveanon:3172kB activefile:16kB inactivefile:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slabreclaimable:28kB slabunreclaimable:16kB kernelstack:72kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

9 Time(s): DMA per-cpu:

2 Time(s): DMA: 3624kB 28kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1464kB

1 Time(s): DMA: 3634kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1452kB

1 Time(s): DMA: 3634kB 28kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1468kB

2 Time(s): DMA: 3644kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1456kB

1 Time(s): DMA: 3654kB 68kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1508kB

2 Time(s): DMA: 3994kB 68kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1644kB

9 Time(s): Free swap = 0kB

1 Time(s): Killed process 11485 (apache2)

1 Time(s): Killed process 12216 (apache2)

1 Time(s): Killed process 12329 (apache2)

1 Time(s): Killed process 12802 (apache2)

1 Time(s): Killed process 12821 (python)

1 Time(s): Killed process 13055 (apache2)

1 Time(s): Killed process 14200 (apache2)

1 Time(s): Killed process 14366 (apache2)

9 Time(s): Mem-Info:

1 Time(s): Normal free:2204kB min:2308kB low:2884kB high:3460kB activeanon:166036kB inactiveanon:166152kB activefile:28kB inactivefile:124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:31108kB slabreclaimable:1712kB slabunreclaimable:5360kB kernelstack:776kB pagetables:1888kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:32 allunreclaimable? no

1 Time(s): Normal free:2276kB min:2308kB low:2884kB high:3460kB activeanon:165908kB inactiveanon:166004kB activefile:112kB inactivefile:1144kB unevictable:0kB isolated(anon):4kB isolated(file):120kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:308kB shmem:31108kB slabreclaimable:1668kB slabunreclaimable:5416kB kernelstack:784kB pagetables:1812kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:432 allunreclaimable? yes

1 Time(s): Normal free:2284kB min:2308kB low:2884kB high:3460kB activeanon:166128kB inactiveanon:166128kB activefile:64kB inactivefile:472kB unevictable:0kB isolated(anon):12kB isolated(file):116kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:31108kB slabreclaimable:1868kB slabunreclaimable:5360kB kernelstack:760kB pagetables:1824kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:321 allunreclaimable? yes

1 Time(s): Normal free:2324kB min:2308kB low:2884kB high:3460kB activeanon:166036kB inactiveanon:166152kB activefile:140kB inactivefile:236kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:31108kB slabreclaimable:1712kB slabunreclaimable:5360kB kernelstack:776kB pagetables:1888kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:64 allunreclaimable? no

1 Time(s): Normal free:2668kB min:2308kB low:2884kB high:3460kB activeanon:165028kB inactiveanon:165820kB activefile:64kB inactivefile:56kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:204kB shmem:31108kB slabreclaimable:1848kB slabunreclaimable:5436kB kernelstack:768kB pagetables:1824kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): Normal free:5400kB min:2308kB low:2884kB high:3460kB activeanon:163116kB inactiveanon:164348kB activefile:128kB inactivefile:2580kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:616kB shmem:31108kB slabreclaimable:1776kB slabunreclaimable:5372kB kernelstack:784kB pagetables:1796kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): Normal free:8936kB min:2308kB low:2884kB high:3460kB activeanon:160236kB inactiveanon:166036kB activefile:0kB inactivefile:124kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:184kB shmem:31108kB slabreclaimable:1692kB slabunreclaimable:5400kB kernelstack:776kB pagetables:1920kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): Normal free:9360kB min:2308kB low:2884kB high:3460kB activeanon:161744kB inactiveanon:163080kB activefile:112kB inactivefile:904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:164kB shmem:31108kB slabreclaimable:1680kB slabunreclaimable:5452kB kernelstack:832kB pagetables:1820kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): Normal free:9552kB min:2308kB low:2884kB high:3460kB activeanon:159116kB inactiveanon:166036kB activefile:60kB inactivefile:4kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:184kB shmem:31108kB slabreclaimable:1692kB slabunreclaimable:5400kB kernelstack:776kB pagetables:1920kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

9 Time(s): Normal per-cpu:

1 Time(s): Normal: 11504kB 338kB 3016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 5344kB

1 Time(s): Normal: 20684kB 798kB 216kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 8936kB

1 Time(s): Normal: 20764kB 1358kB 216kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 9416kB

1 Time(s): Normal: 21934kB 1088kB 216kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 9668kB

2 Time(s): Normal: 5584kB 28kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2264kB

1 Time(s): Normal: 5664kB 08kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2280kB

1 Time(s): Normal: 5814kB 08kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2340kB

1 Time(s): Normal: 6854kB 48kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2788kB

1 Time(s): Out of memory: kill process 11485 (apache2) score 25572 or a child

1 Time(s): Out of memory: kill process 12216 (apache2) score 25641 or a child

1 Time(s): Out of memory: kill process 12329 (apache2) score 25574 or a child

1 Time(s): Out of memory: kill process 12800 (apache2) score 25791 or a child

1 Time(s): Out of memory: kill process 12802 (apache2) score 25699 or a child

1 Time(s): Out of memory: kill process 13055 (apache2) score 25315 or a child

1 Time(s): Out of memory: kill process 14200 (apache2) score 25704 or a child

1 Time(s): Out of memory: kill process 14366 (apache2) score 22787 or a child

1 Time(s): Pid: 11485, comm: apache2 Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 12215, comm: apache2 Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 12552, comm: apache2 Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 12821, comm: python Not tainted 2.6.32-linode23 #1

2 Time(s): Pid: 12843, comm: maxlifetime Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 14366, comm: apache2 Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 1813, comm: fail2ban-server Not tainted 2.6.32-linode23 #1

1 Time(s): Pid: 2661, comm: mountall Not tainted 2.6.32-linode23 #1

9 Time(s): Swap cache stats: add 0, delete 0, find 0/0

9 Time(s): Total swap = 0kB

1 Time(s): [] ? xen_clts+0x46/0x50

1 Time(s): [] ? xenmcflush+0xf7/0x1b0

4 Time(s): [] ? xensetpte_at+0x80/0xf0

3 Time(s): [] ? xenforceevtchn_callback+0x17/0x30

3 Time(s): [] ? xenrestorefldirectend+0x0/0x1

5 Time(s): [] ? check_events+0x8/0xc

4 Time(s): [] ? sys_clone+0x35/0x50

4 Time(s): [] ? syscall_call+0x7/0xb

2 Time(s): [] ? handle_irq+0x17/0x20

10 Time(s): [] ? dopagefault+0x0/0x330

7 Time(s): [] ? dopagefault+0x134/0x330

1 Time(s): [] ? finishtaskswitch+0x47/0xa0

4 Time(s): [] ? copy_process+0xba/0xfd0

4 Time(s): [] ? do_fork+0x70/0x2e0

2 Time(s): [] ? irq_exit+0x31/0x70

1 Time(s): [] ? posixktimeget_ts+0x0/0x10

1 Time(s): [] ? _wakeup_bit+0x2e/0x40

2 Time(s): [] ? hrtimergetres+0x0/0x30

1 Time(s): [] ? ktimegetts+0xe3/0x100

5 Time(s): [] ? filemap_fault+0x397/0x3a0

8 Time(s): [] ? oomkillprocess+0x9a/0x280

8 Time(s): [] ? _outof_memory+0xfc/0x160

8 Time(s): [] ? outofmemory+0x54/0xb0

1 Time(s): [] ? _allocpages_nodemask+0x49c/0x580

8 Time(s): [] ? _allocpages_nodemask+0x561/0x580

4 Time(s): [] ? _getfree_pages+0x14/0x30

5 Time(s): [] ? _dopagecachereadahead+0xdd/0x1f0

5 Time(s): [] ? ra_submit+0x27/0x40

5 Time(s): [] ? _dofault+0x5a/0x490

5 Time(s): [] ? handlemmfault+0x167/0x990

1 Time(s): [] ? mmap_region+0x1d9/0x480

1 Time(s): [] ? pollselectcopy_remaining+0xd9/0x120

1 Time(s): [] ? sys_select+0x40/0xb0

2 Time(s): [] ? unmask_evtchn+0x1c/0xe0

4 Time(s): [] ? doblkifrequest+0x24/0x340

1 Time(s): [] ? spinunlock_irqrestore+0x1e/0x30

1 Time(s): activeanon:40578 inactiveanon:42302 isolated_anon:1

1 Time(s): activeanon:40802 inactiveanon:42302 isolated_anon:1

1 Time(s): activeanon:41189 inactiveanon:41579 isolated_anon:0

1 Time(s): activeanon:41550 inactiveanon:41916 isolated_anon:1

1 Time(s): activeanon:42048 inactiveanon:42227 isolated_anon:0

1 Time(s): activeanon:42207 inactiveanon:42311 isolated_anon:1

2 Time(s): activeanon:42294 inactiveanon:42331 isolated_anon:0

1 Time(s): activeanon:42336 inactiveanon:42304 isolated_anon:3

1 Time(s): activefile:0 inactivefile:31 isolated_file:0

1 Time(s): activefile:13 inactivefile:43 isolated_file:0

1 Time(s): activefile:18 inactivefile:43 isolated_file:0

1 Time(s): activefile:19 inactivefile:1 isolated_file:0

1 Time(s): activefile:26 inactivefile:118 isolated_file:29

1 Time(s): activefile:28 inactivefile:336 isolated_file:30

1 Time(s): activefile:29 inactivefile:226 isolated_file:0

1 Time(s): activefile:33 inactivefile:648 isolated_file:0

1 Time(s): activefile:41 inactivefile:71 isolated_file:0

2 Time(s): apache2 invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0

2 Time(s): apache2 invoked oom-killer: gfpmask=0xd0, order=1, oomadj=0

1 Time(s): fail2ban-server invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0

1 Time(s): free:1031 slabreclaimable:473 slabunreclaimable:1363

1 Time(s): free:1714 slabreclaimable:451 slabunreclaimable:1346

1 Time(s): free:2645 slabreclaimable:430 slabunreclaimable:1354

1 Time(s): free:2717 slabreclaimable:428 slabunreclaimable:1366

1 Time(s): free:2769 slabreclaimable:430 slabunreclaimable:1354

1 Time(s): free:917 slabreclaimable:435 slabunreclaimable:1343

1 Time(s): free:934 slabreclaimable:483 slabunreclaimable:1343

1 Time(s): free:936 slabreclaimable:425 slabunreclaimable:1360

1 Time(s): free:947 slabreclaimable:435 slabunreclaimable:1343

9 Time(s): lowmem_reserve[]: 0 0 0 0

9 Time(s): lowmem_reserve[]: 0 341 341 341

1 Time(s): mapped:158 shmem:7777 pagetables:452 bounce:0

1 Time(s): mapped:33 shmem:7777 pagetables:459 bounce:0

2 Time(s): mapped:39 shmem:7777 pagetables:475 bounce:0

1 Time(s): mapped:42 shmem:7777 pagetables:480 bounce:0

2 Time(s): mapped:50 shmem:7777 pagetables:483 bounce:0

1 Time(s): mapped:60 shmem:7777 pagetables:459 bounce:0

1 Time(s): mapped:86 shmem:7777 pagetables:461 bounce:0

2 Time(s): maxlifetime invoked oom-killer: gfpmask=0xd0, order=1, oomadj=0

1 Time(s): mountall invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0

1 Time(s): python: page allocation failure. order:0, mode:0x201da

9 Time(s): unevictable:0 dirty:0 writeback:0 unstable:0