Apache and disk IO rate warning

Question

Apache and disk IO rate warning

general

Hello everyone!

Today I received the following
> Your Linode, hephaestus, has exceeded the notification threshold (1000) for disk io rate by averaging 1003.72 for the last 2 hours.
So I started looking around. Everything seemed as usually except weird messages in /var/log/syslog:

Dec 13 14:01:05 hephaestus kernel: DMA free:2092kB min:84kB low:104kB high:124kB active_anon:1340kB inactive_anon:1408kB active_file:4kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:224kB mapped:0kB shmem:0kB slab_reclaimable:184kB slab_unreclaimable:520kB kernel_stack:320kB pagetables:640kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:76 all_unreclaimable? yes
Dec 13 14:01:05 hephaestus kernel: lowmem_reserve[]: 0 500 500 500
Dec 13 14:01:05 hephaestus kernel: Normal free:2800kB min:2816kB low:3520kB high:4224kB active_anon:231656kB inactive_anon:231744kB active_file:212kB inactive_file:1448kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:512064kB mlocked:0kB dirty:0kB writeback:204kB mapped:192kB shmem:28kB slab_reclaimable:4812kB slab_unreclaimable:11584kB kernel_stack:816kB pagetables:5124kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2537 all_unreclaimable? yes
Dec 13 14:01:05 hephaestus kernel: lowmem_reserve[]: 0 0 0 0
Dec 13 14:01:05 hephaestus kernel: DMA: 370*4kB 77*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2096kB
Dec 13 14:01:05 hephaestus kernel: Normal: 700*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2800kB
Dec 13 14:01:05 hephaestus kernel: 8706 total pagecache pages
Dec 13 14:01:05 hephaestus kernel: 8164 pages in swap cache
Dec 13 14:01:05 hephaestus kernel: Swap cache stats: add 424580, delete 416416, find 1746509/1776061
Dec 13 14:01:05 hephaestus kernel: Free swap  = 0kB
Dec 13 14:01:05 hephaestus kernel: Total swap = 262140kB
Dec 13 14:01:05 hephaestus kernel: 133104 pages RAM
Dec 13 14:01:05 hephaestus kernel: 0 pages HighMem
Dec 13 14:01:05 hephaestus kernel: 5833 pages reserved
Dec 13 14:01:05 hephaestus kernel: 13133 pages shared
Dec 13 14:01:05 hephaestus kernel: 124767 pages non-shared
Dec 13 14:01:05 hephaestus kernel: Out of memory: Kill process 18793 (httpd) score 26 or sacrifice child
Dec 13 14:01:05 hephaestus kernel: Killed process 18793 (httpd) total-vm:77020kB, anon-rss:11152kB, file-rss:0kB
Dec 13 14:01:22 hephaestus kernel: httpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Dec 13 14:01:22 hephaestus kernel: Pid: 21631, comm: httpd Not tainted 3.0.4-linode38 #1
Dec 13 14:01:22 hephaestus kernel: Call Trace:
Dec 13 14:01:22 hephaestus kernel:  [<c0187ee6>] ? T.677+0x76/0x1b0
Dec 13 14:01:22 hephaestus kernel:  [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Dec 13 14:01:22 hephaestus kernel:  [<c06bf8b1>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Dec 13 14:01:22 hephaestus kernel:  [<c045c387>] ? ___ratelimit+0x97/0x110
Dec 13 14:01:22 hephaestus kernel:  [<c0178ae4>] ? __delayacct_freepages_end+0x24/0x30
Dec 13 14:01:22 hephaestus kernel:  [<c018807e>] ? T.676+0x5e/0x150
Dec 13 14:01:22 hephaestus kernel:  [<c013e25c>] ? has_capability_noaudit+0xc/0x20
Dec 13 14:01:22 hephaestus kernel:  [<c0187d65>] ? oom_badness+0xb5/0x110
Dec 13 14:01:22 hephaestus kernel:  [<c01883aa>] ? out_of_memory+0x23a/0x300
Dec 13 14:01:22 hephaestus kernel:  [<c018bccb>] ? __alloc_pages_nodemask+0x61b/0x630
Dec 13 14:01:22 hephaestus kernel:  [<c01ab487>] ? read_swap_cache_async+0xb7/0xf0
Dec 13 14:01:22 hephaestus kernel:  [<c01ab521>] ? swapin_readahead+0x61/0x90
Dec 13 14:01:22 hephaestus kernel:  [<c019dca0>] ? do_swap_page+0x4a0/0x5f0
Dec 13 14:01:22 hephaestus kernel:  [<c019e02c>] ? handle_pte_fault+0x23c/0x2f0
Dec 13 14:01:22 hephaestus kernel:  [<c019e5f4>] ? handle_mm_fault+0xf4/0x1b0
Dec 13 14:01:22 hephaestus kernel:  [<c011f261>] ? do_page_fault+0x101/0x3a0
Dec 13 14:01:22 hephaestus kernel:  [<c017580b>] ? handle_edge_irq+0x6b/0xf0
Dec 13 14:01:22 hephaestus kernel:  [<c04770de>] ? __xen_evtchn_do_upcall+0x1ce/0x210
Dec 13 14:01:22 hephaestus kernel:  [<c0105c27>] ? xen_force_evtchn_callback+0x17/0x30
Dec 13 14:01:22 hephaestus kernel:  [<c0106404>] ? check_events+0x8/0xc
Dec 13 14:01:22 hephaestus kernel:  [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Dec 13 14:01:22 hephaestus kernel:  [<c017793e>] ? rcu_enter_nohz+0x2e/0xb0
Dec 13 14:01:22 hephaestus kernel:  [<c011f160>] ? mm_fault_error+0x130/0x130
Dec 13 14:01:22 hephaestus kernel:  [<c06c0106>] ? error_code+0x5a/0x60
Dec 13 14:01:22 hephaestus kernel:  [<c06b0000>] ? sctp_err_lookup+0x90/0x110
Dec 13 14:01:22 hephaestus kernel:  [<c011f160>] ? mm_fault_error+0x130/0x130
Dec 13 14:01:22 hephaestus kernel: Mem-Info:
Dec 13 14:01:22 hephaestus kernel: DMA per-cpu:
Dec 13 14:01:22 hephaestus kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 13 14:01:22 hephaestus kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec 13 14:01:22 hephaestus kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec 13 14:01:22 hephaestus kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec 13 14:01:22 hephaestus kernel: Normal per-cpu:
Dec 13 14:01:22 hephaestus kernel: CPU    0: hi:  186, btch:  31 usd:   1
Dec 13 14:01:22 hephaestus kernel: CPU    1: hi:  186, btch:  31 usd:  62
Dec 13 14:01:22 hephaestus kernel: CPU    2: hi:  186, btch:  31 usd:  32
Dec 13 14:01:22 hephaestus kernel: CPU    3: hi:  186, btch:  31 usd:  22
Dec 13 14:01:22 hephaestus kernel: active_anon:58281 inactive_anon:58387 isolated_anon:0
Dec 13 14:01:22 hephaestus kernel:  active_file:213 inactive_file:623 isolated_file:0
Dec 13 14:01:22 hephaestus kernel:  unevictable:0 dirty:0 writeback:6 unstable:0
Dec 13 14:01:22 hephaestus kernel:  free:1249 slab_reclaimable:1177 slab_unreclaimable:2986
Dec 13 14:01:22 hephaestus kernel:  mapped:200 shmem:7 pagetables:1397 bounce:0</c011f160></c06b0000></c06c0106></c011f160></c017793e></c01063fb></c0106404></c0105c27></c04770de></c017580b></c011f261></c019e5f4></c019e02c></c019dca0></c01ab521></c01ab487></c018bccb></c01883aa></c0187d65></c013e25c></c018807e></c0178ae4></c045c387></c06bf8b1></c01063fb></c0187ee6>

I don't know what to do with them as I only recently started looking into webservers, etc. As far as I understand then apache (httpd) is trashing my swap (Free swap = 0kB)? I checked apache logs and everything was as usual - couple of bots, people looking for phpmyadmin, etc. Nothing big.

free -m

             total       used       free     shared    buffers     cached
Mem:           497        323        173          0         20         87
-/+ buffers/cache:        215        281
Swap:          255         13        242

Im running slackware 13.37.0 with
* apache/2.2.17 prefork

php 5.3.6 (cli)
mysql 5.1.56

Any help appreciated!

4 Replies

forum:kannary100 · Answer 1 · Dec. 13, 2011, 6:17 p.m.

forum:kannary100 13 years ago

Today I have received two notifications within the last 6 hours on the same issue. Very odd because I know the domains I am running do not usually have that much traffic.

I have more than one domain running and I am trying to figure out why this is happening.

Does anyone knows how how to check which domain is getting the spikes?

forum:obs · Answer 2 · Dec. 13, 2011, 7:53 p.m.

forum:obs 13 years ago

Search the forums for "apache max clients" you'll find a ton of threads, basically apache's defaults aren't suited for small linodes, dropping the maxclients value down to say 5 then working up slowly until you reach a level you're comfortable with.

As for which site, it doesn't matter if you had 10 requests at once and each one takes up 64mb of ram then you'll end up swapping on a linode 512.

forum:r00m · Answer 3 · Dec. 15, 2011, 10:03 a.m.

forum:r00m 13 years ago

Thank you! As far as I understood I need to tune apache and it will make my problem go away.

forum:obs · Answer 4 · Dec. 15, 2011, 10:08 a.m.

forum:obs 13 years ago

Yep pretty much.

Compute

Storage

Networking

Databases

Services

Developer Tools

Industries

Pricing

Community

Engage With Us

Apache and disk IO rate warning

4 Replies

Reply

Tips: