Kernel errors - unsure how to address

Hi -

Logwatch sent me this. I know MySQL crashed yesterday. Not sure why it or Apache would OOM. Also not sure what role kernel errors play here. Thanks very much for any assistance.

(Ubuntu 12.04.1 LTS)

> –------------------- Kernel Begin ------------------------

WARNING: Kernel Errors Present

[] ? mmfaulterror+0x160/0x160 …: 16 Time(s)

[] ? error_code+0x5a/0x60 …: 8 Time(s)

2 Time(s): cache: size-32768, object size: 32768, order: 3

1 Time(s): node 0: slabs: 1/1, objs: 1/1, free: 0

1 Time(s): node 0: slabs: 2/2, objs: 2/2, free: 0

2 Time(s): /usr/share/webm cpuset=/ mems_allowed=0

2 Time(s): /usr/share/webm invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0, oomscoreadj=0

11 Time(s): 0 pages HighMem

1 Time(s): 10020 pages shared

1 Time(s): 104736 pages non-shared

1 Time(s): 11631 pages in swap cache

1 Time(s): 117506 pages non-shared

1 Time(s): 123031 pages non-shared

1 Time(s): 123498 pages non-shared

1 Time(s): 123704 pages non-shared

1 Time(s): 12402 total pagecache pages

1 Time(s): 124207 pages non-shared

1 Time(s): 124326 pages non-shared

1 Time(s): 124363 pages non-shared

1 Time(s): 124461 pages non-shared

1 Time(s): 124515 pages non-shared

1 Time(s): 124911 pages non-shared

11 Time(s): 133104 pages RAM

1 Time(s): 1762 pages in swap cache

1 Time(s): 18484 total pagecache pages

1 Time(s): 2296 pages in swap cache

1 Time(s): 2447 total pagecache pages

1 Time(s): 2501 pages in swap cache

1 Time(s): 28879 pages shared

1 Time(s): 2936 total pagecache pages

1 Time(s): 3071 pages in swap cache

1 Time(s): 3325 total pagecache pages

1 Time(s): 3517 total pagecache pages

1 Time(s): 37423 pages shared

1 Time(s): 44325 total pagecache pages

1 Time(s): 4583 pages in swap cache

1 Time(s): 4758 pages in swap cache

1 Time(s): 5200 pages shared

1 Time(s): 5249 pages shared

1 Time(s): 5412 total pagecache pages

1 Time(s): 5606 pages in swap cache

11 Time(s): 5960 pages reserved

1 Time(s): 6113 pages shared

1 Time(s): 6283 pages shared

1 Time(s): 6493 pages in swap cache

1 Time(s): 6496 pages shared

1 Time(s): 6650 pages shared

1 Time(s): 6670 pages in swap cache

1 Time(s): 6886 pages shared

1 Time(s): 7297 pages in swap cache

1 Time(s): 7629 total pagecache pages

1 Time(s): 7676 total pagecache pages

1 Time(s): 7898 pages shared

1 Time(s): 8216 total pagecache pages

11 Time(s): CPU 0: hi: 0, btch: 1 usd: 0

4 Time(s): CPU 0: hi: 186, btch: 31 usd: 0

1 Time(s): CPU 0: hi: 186, btch: 31 usd: 4

1 Time(s): CPU 0: hi: 186, btch: 31 usd: 24

2 Time(s): CPU 0: hi: 186, btch: 31 usd: 30

1 Time(s): CPU 0: hi: 186, btch: 31 usd: 42

1 Time(s): CPU 0: hi: 186, btch: 31 usd: 58

1 Time(s): CPU 0: hi: 186, btch: 31 usd: 113

11 Time(s): CPU 1: hi: 0, btch: 1 usd: 0

6 Time(s): CPU 1: hi: 186, btch: 31 usd: 0

1 Time(s): CPU 1: hi: 186, btch: 31 usd: 27

1 Time(s): CPU 1: hi: 186, btch: 31 usd: 30

1 Time(s): CPU 1: hi: 186, btch: 31 usd: 32

1 Time(s): CPU 1: hi: 186, btch: 31 usd: 59

1 Time(s): CPU 1: hi: 186, btch: 31 usd: 125

11 Time(s): CPU 2: hi: 0, btch: 1 usd: 0

3 Time(s): CPU 2: hi: 186, btch: 31 usd: 0

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 1

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 5

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 27

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 30

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 35

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 80

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 133

1 Time(s): CPU 2: hi: 186, btch: 31 usd: 156

11 Time(s): CPU 3: hi: 0, btch: 1 usd: 0

5 Time(s): CPU 3: hi: 186, btch: 31 usd: 0

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 3

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 7

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 25

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 30

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 115

1 Time(s): CPU 3: hi: 186, btch: 31 usd: 158

11 Time(s): Call Trace:

1 Time(s): DMA free:2044kB min:84kB low:104kB high:124kB activeanon:1276kB inactiveanon:1588kB activefile:1240kB inactivefile:592kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:716kB shmem:1460kB slabreclaimable:48kB slabunreclaimable:24kB kernelstack:8kB pagetables:0kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): DMA free:2056kB min:84kB low:104kB high:124kB activeanon:1464kB inactiveanon:1588kB activefile:1300kB inactivefile:344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:716kB shmem:1460kB slabreclaimable:48kB slabunreclaimable:20kB kernelstack:8kB pagetables:0kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:12 allunreclaimable? no

1 Time(s): DMA free:2080kB min:84kB low:104kB high:124kB activeanon:2040kB inactiveanon:2456kB activefile:0kB inactivefile:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:812kB slabreclaimable:40kB slabunreclaimable:72kB kernelstack:24kB pagetables:20kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:165 allunreclaimable? yes

1 Time(s): DMA free:2080kB min:84kB low:104kB high:124kB activeanon:2128kB inactiveanon:2304kB activefile:8kB inactivefile:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:12kB shmem:640kB slabreclaimable:40kB slabunreclaimable:60kB kernelstack:88kB pagetables:24kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:141 allunreclaimable? yes

1 Time(s): DMA free:2080kB min:84kB low:104kB high:124kB activeanon:2148kB inactiveanon:2220kB activefile:0kB inactivefile:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:560kB slabreclaimable:40kB slabunreclaimable:172kB kernelstack:96kB pagetables:24kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

1 Time(s): DMA free:2080kB min:84kB low:104kB high:124kB activeanon:2192kB inactiveanon:2304kB activefile:0kB inactivefile:92kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:1480kB slabreclaimable:60kB slabunreclaimable:24kB kernelstack:40kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

1 Time(s): DMA free:2080kB min:84kB low:104kB high:124kB activeanon:2204kB inactiveanon:2348kB activefile:0kB inactivefile:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:1480kB slabreclaimable:52kB slabunreclaimable:24kB kernelstack:32kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? yes

1 Time(s): DMA free:2084kB min:84kB low:104kB high:124kB activeanon:2104kB inactiveanon:2140kB activefile:52kB inactivefile:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:32kB shmem:696kB slabreclaimable:40kB slabunreclaimable:156kB kernelstack:88kB pagetables:16kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:455 allunreclaimable? yes

1 Time(s): DMA free:2084kB min:84kB low:104kB high:124kB activeanon:2144kB inactiveanon:2304kB activefile:4kB inactivefile:0kB unevictable:0kB isolated(anon):112kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:664kB slabreclaimable:40kB slabunreclaimable:28kB kernelstack:80kB pagetables:20kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:103 allunreclaimable? yes

1 Time(s): DMA free:2084kB min:84kB low:104kB high:124kB activeanon:2244kB inactiveanon:2284kB activefile:0kB inactivefile:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:812kB slabreclaimable:52kB slabunreclaimable:24kB kernelstack:24kB pagetables:44kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:60 allunreclaimable? yes

1 Time(s): DMA free:2108kB min:84kB low:104kB high:124kB activeanon:2088kB inactiveanon:2140kB activefile:32kB inactivefile:264kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:36kB shmem:1444kB slabreclaimable:44kB slabunreclaimable:52kB kernelstack:8kB pagetables:12kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

11 Time(s): DMA per-cpu:

1 Time(s): DMA: 14kB 178kB 4016kB 3532kB 364kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2092kB

1 Time(s): DMA: 14kB 38kB 3316kB 3432kB 764kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2092kB

2 Time(s): DMA: 14kB 98kB 4916kB 3332kB 264kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2044kB

1 Time(s): DMA: 114kB 248kB 4216kB 3732kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2092kB

1 Time(s): DMA: 194kB 398kB 3216kB 2532kB 664kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2084kB

1 Time(s): DMA: 254kB 268kB 4316kB 3432kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2084kB

1 Time(s): DMA: 34kB 168kB 2216kB 3232kB 964kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2092kB

1 Time(s): DMA: 54kB 288kB 3116kB 2632kB 864kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2084kB

1 Time(s): DMA: 74kB 158kB 1116kB 3432kB 1164kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2116kB

1 Time(s): DMA: 74kB 198kB 4116kB 3732kB 164kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2084kB

9 Time(s): Free swap = 0kB

1 Time(s): Free swap = 118344kB

1 Time(s): Free swap = 137272kB

1 Time(s): Killed process 29433 (apache2) total-vm:102384kB, anon-rss:52932kB, file-rss:0kB

1 Time(s): Killed process 29528 (apache2) total-vm:191540kB, anon-rss:133988kB, file-rss:0kB

1 Time(s): Killed process 29902 (apache2) total-vm:102336kB, anon-rss:22324kB, file-rss:0kB

1 Time(s): Killed process 29977 (mysqld) total-vm:333508kB, anon-rss:9664kB, file-rss:0kB

1 Time(s): Killed process 30143 (apache2) total-vm:102336kB, anon-rss:41896kB, file-rss:0kB

1 Time(s): Killed process 30212 (apache2) total-vm:103060kB, anon-rss:38388kB, file-rss:0kB

1 Time(s): Killed process 30219 (apache2) total-vm:110128kB, anon-rss:54168kB, file-rss:0kB

1 Time(s): Killed process 30221 (apache2) total-vm:104692kB, anon-rss:41108kB, file-rss:24kB

1 Time(s): Killed process 4524 (mysqld) total-vm:348068kB, anon-rss:26300kB, file-rss:0kB

11 Time(s): Mem-Info:

1 Time(s): Normal free:2704kB min:2816kB low:3520kB high:4224kB activeanon:219192kB inactiveanon:219476kB activefile:744kB inactivefile:1108kB unevictable:0kB isolated(anon):52kB isolated(file):68kB present:512064kB mlocked:0kB dirty:0kB writeback:0kB mapped:684kB shmem:108kB slabreclaimable:28080kB slabunreclaimable:11688kB kernelstack:1808kB pagetables:5324kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:2945 allunreclaimable? yes

1 Time(s): Normal free:2720kB min:2816kB low:3520kB high:4224kB activeanon:219676kB inactiveanon:219832kB activefile:584kB inactivefile:996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:0kB mapped:724kB shmem:784kB slabreclaimable:27616kB slabunreclaimable:11516kB kernelstack:1800kB pagetables:5420kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:2545 allunreclaimable? yes

1 Time(s): Normal free:2732kB min:2816kB low:3520kB high:4224kB activeanon:219792kB inactiveanon:219900kB activefile:372kB inactivefile:572kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:168kB mapped:744kB shmem:488kB slabreclaimable:27748kB slabunreclaimable:11376kB kernelstack:1792kB pagetables:5368kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:1391 allunreclaimable? yes

1 Time(s): Normal free:2772kB min:2816kB low:3520kB high:4224kB activeanon:219004kB inactiveanon:219144kB activefile:1012kB inactivefile:1312kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:108kB mapped:1156kB shmem:976kB slabreclaimable:27464kB slabunreclaimable:11496kB kernelstack:1776kB pagetables:5792kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:3685 allunreclaimable? yes

1 Time(s): Normal free:2772kB min:2816kB low:3520kB high:4224kB activeanon:219232kB inactiveanon:219240kB activefile:548kB inactivefile:812kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:0kB mapped:540kB shmem:972kB slabreclaimable:27556kB slabunreclaimable:11492kB kernelstack:1760kB pagetables:5680kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:2217 allunreclaimable? yes

1 Time(s): Normal free:2780kB min:2816kB low:3520kB high:4224kB activeanon:220276kB inactiveanon:220284kB activefile:36kB inactivefile:152kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:512064kB mlocked:0kB dirty:4kB writeback:0kB mapped:276kB shmem:764kB slabreclaimable:27468kB slabunreclaimable:11608kB kernelstack:1744kB pagetables:5612kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:529 allunreclaimable? yes

1 Time(s): Normal free:2788kB min:2816kB low:3520kB high:4224kB activeanon:219700kB inactiveanon:219920kB activefile:524kB inactivefile:648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:12kB mapped:604kB shmem:976kB slabreclaimable:27540kB slabunreclaimable:11544kB kernelstack:1744kB pagetables:5560kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:1851 allunreclaimable? yes

1 Time(s): Normal free:2816kB min:2816kB low:3520kB high:4224kB activeanon:220008kB inactiveanon:221364kB activefile:8kB inactivefile:108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:8kB mapped:208kB shmem:680kB slabreclaimable:28500kB slabunreclaimable:11112kB kernelstack:1712kB pagetables:5080kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:201 allunreclaimable? yes

1 Time(s): Normal free:3888kB min:2816kB low:3520kB high:4224kB activeanon:220412kB inactiveanon:220436kB activefile:68kB inactivefile:184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:88kB mapped:316kB shmem:688kB slabreclaimable:28348kB slabunreclaimable:11184kB kernelstack:1464kB pagetables:4976kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:0 allunreclaimable? no

1 Time(s): Normal free:4216kB min:2816kB low:3520kB high:4224kB activeanon:195832kB inactiveanon:196092kB activefile:25712kB inactivefile:21644kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:68kB writeback:24kB mapped:15772kB shmem:1132kB slabreclaimable:29448kB slabunreclaimable:11096kB kernelstack:1656kB pagetables:4412kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:82 allunreclaimable? no

1 Time(s): Normal free:4312kB min:2816kB low:3520kB high:4224kB activeanon:149632kB inactiveanon:149808kB activefile:77108kB inactivefile:62460kB unevictable:0kB isolated(anon):0kB isolated(file):108kB present:512064kB mlocked:0kB dirty:108kB writeback:0kB mapped:13568kB shmem:16068kB slabreclaimable:29880kB slabunreclaimable:10660kB kernelstack:1608kB pagetables:3964kB unstable:0kB bounce:0kB writebacktmp:0kB pagesscanned:49 allunreclaimable? no

11 Time(s): Normal per-cpu:

1 Time(s): Normal: 5664kB 498kB 316kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2704kB

1 Time(s): Normal: 6014kB 828kB 8616kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 4436kB

1 Time(s): Normal: 6284kB 288kB 816kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2864kB

1 Time(s): Normal: 6474kB 308kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2844kB

1 Time(s): Normal: 6484kB 148kB 116kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2720kB

1 Time(s): Normal: 6514kB 218kB 216kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2804kB

1 Time(s): Normal: 6674kB 158kB 316kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2836kB

1 Time(s): Normal: 6894kB 38kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2780kB

1 Time(s): Normal: 6914kB 18kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 2772kB

1 Time(s): Normal: 7944kB 1308kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 4216kB

1 Time(s): Normal: 8784kB 418kB 316kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 3888kB

1 Time(s): Out of memory: Kill process 29433 (apache2) score 73 or sacrifice child

1 Time(s): Out of memory: Kill process 29528 (apache2) score 176 or sacrifice child

1 Time(s): Out of memory: Kill process 29902 (apache2) score 73 or sacrifice child

1 Time(s): Out of memory: Kill process 29977 (mysqld) score 71 or sacrifice child

1 Time(s): Out of memory: Kill process 30143 (apache2) score 73 or sacrifice child

1 Time(s): Out of memory: Kill process 30212 (apache2) score 71 or sacrifice child

1 Time(s): Out of memory: Kill process 30219 (apache2) score 79 or sacrifice child

1 Time(s): Out of memory: Kill process 30221 (apache2) score 75 or sacrifice child

1 Time(s): Out of memory: Kill process 4524 (mysqld) score 110 or sacrifice child

1 Time(s): Pid: 18922, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 23616, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 29528, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 29547, comm: miniserv.pl Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30009, comm: mysqld Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30081, comm: /usr/share/webm Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30136, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30214, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30221, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30227, comm: apache2 Not tainted 3.5.2-linode45 #1

1 Time(s): Pid: 30326, comm: /usr/share/webm Not tainted 3.5.2-linode45 #1

2 Time(s): SLAB: Unable to allocate memory on node 0 (gfp=0x20)

1 Time(s): Swap cache stats: add 1010198, delete 1005440, find 194980520/195074597

1 Time(s): Swap cache stats: add 1121277, delete 1114607, find 195123126/195225106

1 Time(s): Swap cache stats: add 1161697, delete 1159935, find 195141635/195247703

1 Time(s): Swap cache stats: add 1179851, delete 1172554, find 195144902/195252043

1 Time(s): Swap cache stats: add 1221611, delete 1215118, find 195167240/195278999

1 Time(s): Swap cache stats: add 1239860, delete 1237359, find 195185647/195299303

1 Time(s): Swap cache stats: add 1260371, delete 1248740, find 195190747/195306322

1 Time(s): Swap cache stats: add 1280798, delete 1277727, find 195198043/195315915

1 Time(s): Swap cache stats: add 877305, delete 872722, find 192807923/192893729

1 Time(s): Swap cache stats: add 887937, delete 882331, find 193564369/193650979

1 Time(s): Swap cache stats: add 991791, delete 989495, find 194977489/195071092

11 Time(s): Total swap = 262140kB

Breaking in here to ask: what specifically do the below refer to? What am I reading? Errors that happened to a process? And what's different from those above?

> 1 Time(s): [ 1987] 0 1987 712 0 3 0 0 upstart-socket-

1 Time(s): [ 1987] 0 1987 712 12 0 0 0 upstart-socket-

5 Time(s): [ 1987] 0 1987 712 13 0 0 0 upstart-socket-

1 Time(s): [ 1987] 0 1987 712 13 2 0 0 upstart-socket-

1 Time(s): [ 1987] 0 1987 712 24 0 0 0 upstart-socket-

1 Time(s): [ 2075] 0 2075 1672 0 3 -17 -1000 sshd

1 Time(s): [ 2075] 0 2075 1672 1 3 -17 -1000 sshd

1 Time(s): [ 2075] 0 2075 1672 3 1 -17 -1000 sshd

3 Time(s): [ 2075] 0 2075 1672 4 1 -17 -1000 sshd

2 Time(s): [ 2075] 0 2075 1672 11 1 -17 -1000 sshd

1 Time(s): [ 2075] 0 2075 1672 12 1 -17 -1000 sshd

1 Time(s): [ 2085] 101 2085 8071 50 0 0 0 rsyslogd

1 Time(s): [ 2085] 101 2085 8071 55 0 0 0 rsyslogd

2 Time(s): [ 2085] 101 2085 8071 68 0 0 0 rsyslogd

3 Time(s): [ 2085] 101 2085 8071 80 0 0 0 rsyslogd

2 Time(s): [ 2085] 101 2085 8071 81 0 0 0 rsyslogd

9 Time(s): [ 2098] 0 2098 733 0 0 0 0 dhclient3

1 Time(s): [ 2104] 102 2104 816 0 0 0 0 dbus-daemon

1 Time(s): [ 2104] 102 2104 816 6 2 0 0 dbus-daemon

5 Time(s): [ 2104] 102 2104 816 7 2 0 0 dbus-daemon

1 Time(s): [ 2104] 102 2104 816 15 3 0 0 dbus-daemon

1 Time(s): [ 2104] 102 2104 816 23 2 0 0 dbus-daemon

9 Time(s): [ 2190] 0 2190 619 0 0 0 0 atd

2 Time(s): [ 2191] 0 2191 656 15 1 0 0 cron

7 Time(s): [ 2191] 0 2191 656 21 0 0 0 cron

1 Time(s): [ 2244] 103 2244 6117 31 0 0 0 whoopsie

1 Time(s): [ 2244] 103 2244 6117 33 0 0 0 whoopsie

2 Time(s): [ 2244] 103 2244 6117 33 1 0 0 whoopsie

1 Time(s): [ 2244] 103 2244 6117 33 3 0 0 whoopsie

2 Time(s): [ 2244] 103 2244 6117 34 0 0 0 whoopsie

1 Time(s): [ 2244] 103 2244 6117 34 1 0 0 whoopsie

1 Time(s): [ 2244] 103 2244 6117 34 3 0 0 whoopsie

2 Time(s): [ 2681] 0 2681 602 0 0 0 0 getty

7 Time(s): [ 2681] 0 2681 602 1 0 0 0 getty

3 Time(s): [ 2731] 106 2731 1434 29 0 0 0 ntpd

2 Time(s): [ 2731] 106 2731 1434 29 2 0 0 ntpd

1 Time(s): [ 2731] 106 2731 1434 35 0 0 0 ntpd

1 Time(s): [ 2731] 106 2731 1434 36 0 0 0 ntpd

1 Time(s): [ 2731] 106 2731 1434 36 1 0 0 ntpd

1 Time(s): [ 2731] 106 2731 1434 38 1 0 0 ntpd

1 Time(s): [ 4524] 108 4524 87017 6575 1 0 0 mysqld

1 Time(s): [ 5952] 0 5952 1146 11 1 0 0 master

2 Time(s): [ 5952] 0 5952 1146 19 1 0 0 master

2 Time(s): [ 5952] 0 5952 1146 21 0 0 0 master

1 Time(s): [ 5952] 0 5952 1146 22 1 0 0 master

1 Time(s): [ 5952] 0 5952 1146 22 2 0 0 master

1 Time(s): [ 5952] 0 5952 1146 23 2 0 0 master

1 Time(s): [ 5952] 0 5952 1146 23 3 0 0 master

1 Time(s): [ 5954] 107 5954 1163 19 0 0 0 qmgr

5 Time(s): [ 5954] 107 5954 1163 20 0 0 0 qmgr

1 Time(s): [ 5954] 107 5954 1163 23 1 0 0 qmgr

2 Time(s): [ 5954] 107 5954 1163 24 0 0 0 qmgr

5 Time(s): [ 6135] 107 6135 1799 3 0 0 0 tlsmgr

1 Time(s): [ 6135] 107 6135 1799 4 0 0 0 tlsmgr

1 Time(s): [ 6135] 107 6135 1799 25 1 0 0 tlsmgr

1 Time(s): [ 6135] 107 6135 1799 26 1 0 0 tlsmgr

1 Time(s): [ 6135] 107 6135 1799 27 0 0 0 tlsmgr

9 Time(s): [ 7178] 1002 7178 1458 0 0 0 0 screen

1 Time(s): [ 7179] 1002 7179 1862 0 1 0 0 bash

8 Time(s): [ 7179] 1002 7179 1862 1 1 0 0 bash

1 Time(s): [ 7347] 0 7347 4838 178 0 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 195 0 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 196 3 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 241 0 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 250 2 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 317 0 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 332 3 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 339 1 0 0 miniserv.pl

1 Time(s): [ 7347] 0 7347 4838 343 0 0 0 miniserv.pl

2 Time(s): [ 7387] 1002 7387 1860 0 3 0 0 bash

7 Time(s): [ 7387] 1002 7387 1860 1 3 0 0 bash

9 Time(s): [ 7486] 1002 7486 3188 0 1 0 0 vim

9 Time(s): [ pid ] uid tgid totalvm rss cpu oomadj oomscoreadj name

4 Time(s): [11370] 1000 11370 12041 0 0 0 0 memcached

2 Time(s): [11370] 1000 11370 12041 0 1 0 0 memcached

2 Time(s): [11370] 1000 11370 12041 0 2 0 0 memcached

1 Time(s): [11370] 1000 11370 12041 3 1 0 0 memcached

1 Time(s): [11740] 0 11740 710 0 2 0 0 upstart-udev-br

1 Time(s): [11740] 0 11740 710 4 2 0 0 upstart-udev-br

5 Time(s): [11740] 0 11740 710 5 2 0 0 upstart-udev-br

1 Time(s): [11740] 0 11740 710 16 2 0 0 upstart-udev-br

1 Time(s): [11740] 0 11740 710 17 2 0 0 upstart-udev-br

1 Time(s): [11742] 0 11742 711 0 0 -17 -1000 udevd

8 Time(s): [11742] 0 11742 711 1 0 -17 -1000 udevd

2 Time(s): [19800] 1002 19800 1705 0 0 0 0 screen

7 Time(s): [19800] 1002 19800 1705 1 0 0 0 screen

2 Time(s): [19801] 1002 19801 1859 0 3 0 0 bash

7 Time(s): [19801] 1002 19801 1859 1 3 0 0 bash

2 Time(s): [19857] 1002 19857 1860 0 0 0 0 bash

7 Time(s): [19857] 1002 19857 1860 1 0 0 0 bash

2 Time(s): [19963] 1002 19963 3199 0 0 0 0 vim

7 Time(s): [19963] 1002 19963 3199 1 0 0 0 vim

2 Time(s): [21674] 1002 21674 3187 0 2 0 0 vim

7 Time(s): [21674] 1002 21674 3187 1 2 0 0 vim

2 Time(s): [21676] 1002 21676 1859 0 1 0 0 bash

7 Time(s): [21676] 1002 21676 1859 1 1 0 0 bash

2 Time(s): [21782] 1002 21782 1860 0 0 0 0 bash

7 Time(s): [21782] 1002 21782 1860 1 0 0 0 bash

9 Time(s): [22370] 1002 22370 1458 0 2 0 0 screen

9 Time(s): [22371] 1002 22371 1860 0 1 0 0 bash

1 Time(s): [24402] 109 24402 12555 28 0 -13 -900 postgres

1 Time(s): [24402] 109 24402 12555 28 2 -13 -900 postgres

3 Time(s): [24402] 109 24402 12555 29 0 -13 -900 postgres

1 Time(s): [24402] 109 24402 12555 29 2 -13 -900 postgres

1 Time(s): [24402] 109 24402 12555 29 3 -13 -900 postgres

1 Time(s): [24402] 109 24402 12555 30 0 -13 -900 postgres

1 Time(s): [24402] 109 24402 12555 31 0 -13 -900 postgres

2 Time(s): [24404] 109 24404 12586 20 0 0 0 postgres

2 Time(s): [24404] 109 24404 12586 20 3 0 0 postgres

1 Time(s): [24404] 109 24404 12586 21 0 0 0 postgres

2 Time(s): [24404] 109 24404 12586 21 1 0 0 postgres

1 Time(s): [24404] 109 24404 12586 21 2 0 0 postgres

1 Time(s): [24404] 109 24404 12586 33 0 0 0 postgres

1 Time(s): [24405] 109 24405 12555 13 2 0 0 postgres

2 Time(s): [24405] 109 24405 12555 14 0 0 0 postgres

1 Time(s): [24405] 109 24405 12555 14 1 0 0 postgres

4 Time(s): [24405] 109 24405 12555 14 2 0 0 postgres

1 Time(s): [24405] 109 24405 12555 14 3 0 0 postgres

2 Time(s): [24406] 109 24406 12727 100 1 0 0 postgres

2 Time(s): [24406] 109 24406 12727 101 0 0 0 postgres

2 Time(s): [24406] 109 24406 12727 101 3 0 0 postgres

1 Time(s): [24406] 109 24406 12727 114 2 0 0 postgres

1 Time(s): [24406] 109 24406 12727 114 3 0 0 postgres

1 Time(s): [24406] 109 24406 12727 124 0 0 0 postgres

1 Time(s): [24407] 109 24407 5180 64 2 0 0 postgres

1 Time(s): [24407] 109 24407 5180 65 0 0 0 postgres

2 Time(s): [24407] 109 24407 5180 66 0 0 0 postgres

3 Time(s): [24407] 109 24407 5180 66 1 0 0 postgres

1 Time(s): [24407] 109 24407 5180 66 2 0 0 postgres

1 Time(s): [24407] 109 24407 5180 66 3 0 0 postgres

2 Time(s): [24636] 0 24636 1110 0 1 0 0 newrelic-daemon

7 Time(s): [24636] 0 24636 1110 1 1 0 0 newrelic-daemon

8 Time(s): [24638] 0 24638 25485 0 0 0 0 newrelic-daemon

1 Time(s): [24638] 0 24638 25485 0 1 0 0 newrelic-daemon

2 Time(s): [25141] 1002 25141 3184 0 1 0 0 vim

7 Time(s): [25141] 1002 25141 3184 1 1 0 0 vim

2 Time(s): [26701] 1002 26701 1860 0 2 0 0 bash

7 Time(s): [26701] 1002 26701 1860 1 2 0 0 bash

2 Time(s): [27283] 0 27283 2415 0 0 0 0 sshd

7 Time(s): [27283] 0 27283 2415 1 0 0 0 sshd

9 Time(s): [27427] 1000 27427 2415 0 1 0 0 sshd

2 Time(s): [27428] 1000 27428 2592 0 2 0 0 bash

7 Time(s): [27428] 1000 27428 2592 1 2 0 0 bash

1 Time(s): [29086] 107 29086 1151 12 0 0 0 pickup

1 Time(s): [29086] 107 29086 1151 13 0 0 0 pickup

1 Time(s): [29086] 107 29086 1151 13 2 0 0 pickup

2 Time(s): [29086] 107 29086 1151 15 0 0 0 pickup

1 Time(s): [29086] 107 29086 1151 15 1 0 0 pickup

1 Time(s): [29086] 107 29086 1151 17 0 0 0 pickup

1 Time(s): [29086] 107 29086 1151 17 1 0 0 pickup

1 Time(s): [29086] 107 29086 1151 17 2 0 0 pickup

2 Time(s): [29388] 33 29388 26114 12612 0 0 0 apache2

2 Time(s): [29433] 33 29433 25596 3533 1 0 0 apache2

1 Time(s): [29433] 33 29433 25596 13233 1 0 0 apache2

2 Time(s): [29460] 33 29460 21541 256 1 0 0 apache2

1 Time(s): [29501] 33 29501 25589 5479 2 0 0 apache2

1 Time(s): [29501] 33 29501 25589 10279 2 0 0 apache2

1 Time(s): [29510] 33 29510 25584 9398 0 0 0 apache2

1 Time(s): [29510] 33 29510 25584 13230 0 0 0 apache2

1 Time(s): [29516] 33 29516 20390 2247 3 0 0 apache2

1 Time(s): [29516] 33 29516 20390 8497 3 0 0 apache2

1 Time(s): [29524] 33 29524 25530 13420 1 0 0 apache2

1 Time(s): [29524] 33 29524 25723 13577 2 0 0 apache2

2 Time(s): [29526] 33 29526 17055 4902 2 0 0 apache2

1 Time(s): [29528] 33 29528 27243 14579 1 0 0 apache2

1 Time(s): [29528] 33 29528 47885 33497 0 0 0 apache2

2 Time(s): [29529] 33 29529 17055 4902 1 0 0 apache2

1 Time(s): [29532] 33 29532 12942 658 2 0 0 apache2

1 Time(s): [29532] 33 29532 13042 773 1 0 0 apache2

1 Time(s): [29533] 33 29533 12942 658 3 0 0 apache2

1 Time(s): [29533] 33 29533 13042 775 3 0 0 apache2

1 Time(s): [29534] 33 29534 20388 8442 0 0 0 apache2

1 Time(s): [29534] 33 29534 22713 10545 3 0 0 apache2

1 Time(s): [29535] 33 29535 12942 658 1 0 0 apache2

2 Time(s): [29539] 33 29539 12942 658 0 0 0 apache2

2 Time(s): [29542] 33 29542 12942 658 1 0 0 apache2

2 Time(s): [29543] 33 29543 12942 658 2 0 0 apache2

1 Time(s): [29547] 0 29547 4838 266 0 0 0 miniserv.pl

1 Time(s): [29547] 0 29547 5597 2933 2 0 0 /usr/share/webm

1 Time(s): [29614] 0 29614 4838 368 3 0 0 /usr/share/webm

1 Time(s): [29615] 33 29615 12677 354 3 0 0 apache2

1 Time(s): [29633] 33 29633 22191 7998 0 0 0 apache2

1 Time(s): [29633] 33 29633 22191 8279 0 0 0 apache2

1 Time(s): [29633] 33 29633 22191 8873 1 0 0 apache2

1 Time(s): [29633] 33 29633 22191 9028 0 0 0 apache2

1 Time(s): [29633] 33 29633 22255 9827 0 0 0 apache2

1 Time(s): [29633] 33 29633 25270 11775 0 0 0 apache2

1 Time(s): [29633] 33 29633 25399 3383 0 0 0 apache2

1 Time(s): [29636] 33 29636 22228 4138 2 0 0 apache2

1 Time(s): [29636] 33 29636 22228 5774 1 0 0 apache2

1 Time(s): [29636] 33 29636 22228 6570 1 0 0 apache2

1 Time(s): [29636] 33 29636 22228 6592 1 0 0 apache2

1 Time(s): [29636] 33 29636 22228 9031 1 0 0 apache2

1 Time(s): [29636] 33 29636 22228 9062 1 0 0 apache2

1 Time(s): [29636] 33 29636 22228 9671 0 0 0 apache2

1 Time(s): [29640] 0 29640 2415 6 0 0 0 sshd

6 Time(s): [29640] 0 29640 2415 7 0 0 0 sshd

1 Time(s): [29791] 1000 29791 2415 28 0 0 0 sshd

3 Time(s): [29791] 1000 29791 2415 29 0 0 0 sshd

2 Time(s): [29791] 1000 29791 2415 29 1 0 0 sshd

1 Time(s): [29791] 1000 29791 2415 47 1 0 0 sshd

2 Time(s): [29792] 1000 29792 2592 103 3 0 0 bash

1 Time(s): [29792] 1000 29792 2592 104 0 0 0 bash

1 Time(s): [29792] 1000 29792 2592 104 1 0 0 bash

1 Time(s): [29792] 1000 29792 2592 123 3 0 0 bash

1 Time(s): [29792] 1000 29792 2592 132 3 0 0 bash

1 Time(s): [29792] 1000 29792 2592 141 0 0 0 bash

1 Time(s): [29902] 33 29902 25584 5581 2 0 0 apache2

1 Time(s): [29902] 33 29902 25584 6985 2 0 0 apache2

1 Time(s): [29977] 108 29977 83161 2443 0 0 0 mysqld

1 Time(s): [29977] 108 29977 83161 2570 0 0 0 mysqld

1 Time(s): [29977] 108 29977 83377 2416 0 0 0 mysqld

1 Time(s): [29977] 108 29977 83417 2700 0 0 0 mysqld

1 Time(s): [29977] 108 29977 83417 2702 1 0 0 mysqld

1 Time(s): [29977] 108 29977 83417 2720 0 0 0 mysqld

1 Time(s): [29977] 108 29977 83417 2751 0 0 0 mysqld

1 Time(s): [30081] 0 30081 4838 321 1 0 0 miniserv.pl

1 Time(s): [30081] 0 30081 6263 2851 0 0 0 /usr/share/webm

1 Time(s): [30081] 0 30081 6526 3572 0 0 0 /usr/share/webm

1 Time(s): [30081] 0 30081 8546 5251 2 0 0 /usr/share/webm

1 Time(s): [30082] 0 30082 4838 411 1 0 0 miniserv.pl

1 Time(s): [30136] 0 30136 12599 241 0 0 0 apache2

1 Time(s): [30136] 0 30136 12599 241 1 0 0 apache2

1 Time(s): [30136] 0 30136 12599 242 0 0 0 apache2

2 Time(s): [30136] 0 30136 12599 242 1 0 0 apache2

1 Time(s): [30136] 0 30136 12599 250 0 0 0 apache2

2 Time(s): [30136] 0 30136 12599 269 0 0 0 apache2

1 Time(s): [30136] 0 30136 12599 270 0 0 0 apache2

1 Time(s): [30143] 33 30143 25584 10474 0 0 0 apache2

1 Time(s): [30143] 33 30143 25584 10478 1 0 0 apache2

1 Time(s): [30143] 33 30143 25584 11215 1 0 0 apache2

1 Time(s): [30168] 0 30168 4838 86 2 0 0 miniserv.pl

1 Time(s): [30168] 0 30168 4838 92 2 0 0 miniserv.pl

1 Time(s): [30168] 0 30168 4838 200 2 0 0 miniserv.pl

1 Time(s): [30168] 0 30168 5337 2690 3 0 0 /usr/share/webm

I deleted a segment of output to meet Linode forum posting character limit. Full text of logwatch output at http://pastebin.com/weRgNDaJ

> 3 Time(s): [] ? xen_clts+0x44/0x50

2 Time(s): [] ? loadTLSdescriptor+0x36/0x70

4 Time(s): [] ? xenmcflush+0x78/0x110

8 Time(s): [] ? ptepfnto_mfn+0xb5/0xd0

7 Time(s): [] ? xenbatchedset_pte+0x1e/0xf0

3 Time(s): [] ? mathstaterestore+0x20/0xf0

3 Time(s): [] ? mathstaterestore+0xf0/0xf0

8 Time(s): [] ? dopagefault+0x101/0x3a0

1 Time(s): [] ? dopagefault+0x169/0x3a0

9 Time(s): [] ? hasnscapability_noaudit+0xb/0x20

1 Time(s): [] ? modtimerpending+0xba/0x110

1 Time(s): [] ? update_curr+0x202/0x220

1 Time(s): [] ? findbusiestgroup+0x107/0xa00

1 Time(s): [] ? sys_futex+0x76/0x140

3 Time(s): [] ? handleedgeirq+0x6b/0xf0

1 Time(s): [] ? _callrcu+0x94/0x170

1 Time(s): [] ? callrcusched+0x14/0x20

3 Time(s): [] ? rcuirqexit+0x52/0xa0

7 Time(s): [] ? _delayacctfreepages_end+0x24/0x30

1 Time(s): [] ? genericperformwrite+0xc4/0x1e0

1 Time(s): [] ? genericfilebuffered_write+0x5f/0xa0

1 Time(s): [] ? _genericfileaiowrite+0x210/0x500

1 Time(s): [] ? genericfileaio_write+0x74/0xf0

1 Time(s): [] ? grabcachepagewritebegin+0x71/0xe0

8 Time(s): [] ? filemap_fault+0x1f2/0x3c0

9 Time(s): [] ? T.681+0x7d/0x1b0

9 Time(s): [] ? T.680+0x258/0x290

9 Time(s): [] ? outofmemory+0x271/0x320

2 Time(s): [] ? warnallocfailed+0x98/0x100

1 Time(s): [] ? _allocpages_nodemask+0x13f/0x6d0

2 Time(s): [] ? _allocpages_nodemask+0x4d8/0x6d0

9 Time(s): [] ? _allocpages_nodemask+0x6b3/0x6d0

1 Time(s): [] ? vmmmappgoff+0x73/0x90

8 Time(s): [] ? _dofault+0x75/0x550

1 Time(s): [] ? _dofault+0x429/0x550

8 Time(s): [] ? handleptefault+0xa0/0x2f0

8 Time(s): [] ? handlemmfault+0xf5/0x1b0

2 Time(s): [] ? T.933+0x31/0xe0

2 Time(s): [] ? T.930+0x91/0x250

2 Time(s): [] ? cacheallocrefill+0x24e/0x290

2 Time(s): [] ? __kmalloc+0xbe/0xd0

1 Time(s): [] ? dosyncreadv_writev+0xbe/0x100

1 Time(s): [] ? dosyncreadv_writev+0x100/0x100

1 Time(s): [] ? dosyncwrite+0xcd/0x110

2 Time(s): [] ? rwverifyarea+0x66/0x120

1 Time(s): [] ? do_sendfile+0x1be/0x240

1 Time(s): [] ? sys_sendfile+0x98/0xb0

1 Time(s): [] ? doreadvwritev+0xaa/0x1a0

1 Time(s): [] ? vfs_writev+0x3e/0x60

1 Time(s): [] ? sys_writev+0x41/0xa0

1 Time(s): [] ? vfs_write+0xa0/0x170

1 Time(s): [] ? sys_write+0x41/0x80

1 Time(s): [] ? sys_fstat64+0x1e/0x30

1 Time(s): [] ? sys_select+0x40/0xb0

3 Time(s): [] ? splicefrompipe_begin+0x10/0x10

1 Time(s): [] ? pipetosendpage+0x6b/0x90

1 Time(s): [] ? splicefrompipe_feed+0x69/0xf0

1 Time(s): [] ? _splicefrom_pipe+0x4b/0x60

1 Time(s): [] ? splicefrompipe+0x5d/0x80

1 Time(s): [] ? defaultfilesplice_write+0x40/0x40

1 Time(s): [] ? genericsplicesendpage+0x20/0x30

1 Time(s): [] ? dosplicefrom+0x5f/0x90

1 Time(s): [] ? dosplicefrom+0x90/0x90

1 Time(s): [] ? directspliceactor+0x25/0x40

1 Time(s): [] ? splicedirectto_actor+0xbf/0x1d0

1 Time(s): [] ? dosplicedirect+0x58/0x80

1 Time(s): [] ? ext3writebackwrite_end+0xc2/0x140

1 Time(s): [] ? ext3writebegin+0x71/0x220

9 Time(s): [] ? securitycapablenoaudit+0x14/0x20

1 Time(s): [] ? securityfilepermission+0x17/0x90

9 Time(s): [] ? ___ratelimit+0x97/0x110

3 Time(s): [] ? _xenevtchndoupcall+0x1ce/0x210

1 Time(s): [] ? xenevtchndo_upcall+0x1d/0x30

2 Time(s): [] ? addinterruptrandomness+0x2e/0x160

1 Time(s): [] ? kernel_sendpage+0x4e/0x90

1 Time(s): [] ? kernel_sendpage+0x90/0x90

2 Time(s): [] ? sockaiodtor+0x10/0x10

1 Time(s): [] ? sockaiowrite+0x12a/0x180

2 Time(s): [] ? pskbexpandhead+0x48/0x1e0

2 Time(s): [] ? _pskbpull_tail+0x4d/0x2a0

2 Time(s): [] ? netifskbfeatures+0xaf/0xc0

2 Time(s): [] ? devhardstart_xmit+0x1b5/0x3e0

2 Time(s): [] ? devqueuexmit+0xff/0x340

1 Time(s): [] ? eth_header+0x28/0xc0

2 Time(s): [] ? schdirectxmit+0xba/0x180

1 Time(s): [] ? nf_iterate+0x6c/0x90

2 Time(s): [] ? iplocalout+0x18/0x20

2 Time(s): [] ? ipfinishoutput2+0x260/0x260

1 Time(s): [] ? ipfinishoutput+0x16a/0x2e0

1 Time(s): [] ? ip_output+0x94/0xb0

1 Time(s): [] ? tcp_sendmsg+0xa63/0xc20

1 Time(s): [] ? dotcpsendpages+0x4df/0x540

1 Time(s): [] ? dotcpsendpages+0x540/0x540

1 Time(s): [] ? tcp_sendpage+0x7b/0x90

2 Time(s): [] ? tcptransmitskb+0x395/0x660

2 Time(s): [] ? tcp_fragment+0x1ad/0x2e0

2 Time(s): [] ? tcpwritexmit+0x1dd/0x4f0

1 Time(s): [] ? tcppushone+0x2d/0x40

1 Time(s): [] ? _tcppushpendingframes+0x24/0x90

1 Time(s): [] ? inet_sendmsg+0x47/0xb0

1 Time(s): [] ? inetdgramconnect+0x80/0x80

1 Time(s): [] ? inet_sendpage+0x69/0x110

10 Time(s): [] ? sctpv6get_saddr+0x40/0x40

1 Time(s): [] ? rawspin_lock+0x5/0x10

9 Time(s): [] ? rawspinunlockirqrestore+0x11/0x20

3 Time(s): [] ? syscall_call+0x7/0xb

1 Time(s): activeanon:37727 inactiveanon:37849 isolated_anon:0

1 Time(s): activeanon:49324 inactiveanon:49420 isolated_anon:0

1 Time(s): activeanon:55277 inactiveanon:55321 isolated_anon:32

1 Time(s): activeanon:55320 inactiveanon:55404 isolated_anon:13

1 Time(s): activeanon:55344 inactiveanon:55386 isolated_anon:28

1 Time(s): activeanon:55429 inactiveanon:55572 isolated_anon:0

1 Time(s): activeanon:55457 inactiveanon:55556 isolated_anon:0

1 Time(s): activeanon:55509 inactiveanon:55546 isolated_anon:32

1 Time(s): activeanon:55550 inactiveanon:55917 isolated_anon:0

1 Time(s): activeanon:55606 inactiveanon:55626 isolated_anon:32

1 Time(s): activeanon:55654 inactiveanon:55696 isolated_anon:0

1 Time(s): activefile:133 inactivefile:175 isolated_file:0

1 Time(s): activefile:138 inactivefile:203 isolated_file:0

1 Time(s): activefile:146 inactivefile:264 isolated_file:0

1 Time(s): activefile:17 inactivefile:50 isolated_file:0

1 Time(s): activefile:194 inactivefile:343 isolated_file:17

1 Time(s): activefile:19587 inactivefile:15763 isolated_file:0

1 Time(s): activefile:2 inactivefile:50 isolated_file:0

1 Time(s): activefile:266 inactivefile:358 isolated_file:0

1 Time(s): activefile:6753 inactivefile:5497 isolated_file:0

1 Time(s): activefile:9 inactivefile:37 isolated_file:0

1 Time(s): activefile:93 inactivefile:148 isolated_file:0

5 Time(s): apache2 cpuset=/ mems_allowed=0

1 Time(s): apache2 invoked oom-killer: gfpmask=0x10200da, order=0, oomadj=0, oomscoreadj=0

4 Time(s): apache2 invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0, oomscoreadj=0

2 Time(s): apache2: page allocation failure: order:3, mode:0x20

1 Time(s): free:1200 slabreclaimable:6914 slabunreclaimable:2897

1 Time(s): free:1203 slabreclaimable:7031 slabunreclaimable:2935

1 Time(s): free:1204 slabreclaimable:6950 slabunreclaimable:2850

1 Time(s): free:1214 slabreclaimable:6876 slabunreclaimable:2913

1 Time(s): free:1214 slabreclaimable:6899 slabunreclaimable:2880

1 Time(s): free:1215 slabreclaimable:6877 slabunreclaimable:2945

1 Time(s): free:1217 slabreclaimable:6895 slabunreclaimable:2901

1 Time(s): free:1224 slabreclaimable:7140 slabunreclaimable:2784

1 Time(s): free:1492 slabreclaimable:7100 slabunreclaimable:2802

1 Time(s): free:1568 slabreclaimable:7374 slabunreclaimable:2779

1 Time(s): free:1589 slabreclaimable:7482 slabunreclaimable:2671

11 Time(s): lowmem_reserve[]: 0 0 0 0

11 Time(s): lowmem_reserve[]: 0 500 500 500

1 Time(s): mapped:136 shmem:409 pagetables:1425 bounce:0

1 Time(s): mapped:154 shmem:404 pagetables:1396 bounce:0

1 Time(s): mapped:180 shmem:388 pagetables:1334 bounce:0

1 Time(s): mapped:182 shmem:399 pagetables:1360 bounce:0

1 Time(s): mapped:189 shmem:325 pagetables:1353 bounce:0

1 Time(s): mapped:297 shmem:418 pagetables:1452 bounce:0

1 Time(s): mapped:3571 shmem:4382 pagetables:991 bounce:0

1 Time(s): mapped:4122 shmem:648 pagetables:1103 bounce:0

1 Time(s): mapped:60 shmem:540 pagetables:1273 bounce:0

1 Time(s): mapped:71 shmem:331 pagetables:1409 bounce:0

1 Time(s): mapped:87 shmem:542 pagetables:1247 bounce:0

1 Time(s): miniserv.pl cpuset=/ mems_allowed=0

1 Time(s): miniserv.pl invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0, oomscoreadj=0

1 Time(s): mysqld cpuset=/ mems_allowed=0

1 Time(s): mysqld invoked oom-killer: gfpmask=0x201da, order=0, oomadj=0, oomscoreadj=0

3 Time(s): unevictable:0 dirty:0 writeback:0 unstable:0

1 Time(s): unevictable:0 dirty:0 writeback:2 unstable:0

1 Time(s): unevictable:0 dirty:0 writeback:22 unstable:0

1 Time(s): unevictable:0 dirty:0 writeback:28 unstable:0

1 Time(s): unevictable:0 dirty:0 writeback:4 unstable:0

1 Time(s): unevictable:0 dirty:0 writeback:42 unstable:0

1 Time(s): unevictable:0 dirty:1 writeback:0 unstable:0

1 Time(s): unevictable:0 dirty:17 writeback:6 unstable:0

1 Time(s): unevictable:0 dirty:27 writeback:0 unstable:0

–-------------------- Kernel End -------------------------