Munin CPU

@nfn:

Anyone knows why?

What does top or htop show?

James

i can't see the image you posted (it's just not loading) but i think i know what you mean, i have a similar "problem".

munin shows the CPU on a scale from zero to 400% making it hard to see anything meaningful.

Problem is that the linode runs with 4 processors so my guess is that munin simply adds up all the individual percentages (htop shows 4 individual processor indicators as well).

I don't see anything wrong either….

Your munin graph is showing 5-10% CPU utilization for the day, which is normal for websites that don't get hit too hard.

In fact, you should be alarmed if you see anything visually meaningful on that graph, because in that case something is probably consuming more CPU than it should.

Hi, I can't paste top at this time because I'm on my mobile, but I think the problem can be the scale.

@nfn:

… I think the problem can be the scale.

No problem, then. The peak CPU scale is correctly shown as 400%, since your Linode has 4 CPUs, and CPU usage that occasionally reaches - but does not stay - as high as 400% is generally OK at Linode. My CPU usage can sustain 100% or 200% for several hours if someone is surface fitting a large data set, although that does not happen very often.

~~http://pages.zunzun.com/ServerStatistic … sLoad2.png">http://pages.zunzun.com/ServerStatistics/Last24HoursLoad2.png" />

James

http;//zunzun.com~~

No, that graph isn't right, because Munin graphs idle time too. Here's one:

~~http://i44.tinypic.com/2iidhk2.png" />

(not a linode, I wanted something with visible CPU usage!). As you can see, it's an area graph that fills the whole are up to 100*cpus % with system time stacked above idle time stacked above user time.

The "NaN" in the "Idle" line on nfn's graph above is the problem, but from here it's a matter of looking at the munin-node "cpu" plugin to see why it's failing. Here's the output of one of mine:

# /usr/share/munin/plugins/cpu 
user.value 32245294
nice.value 1187
system.value 13282284
idle.value 2151614388
iowait.value 2934277
irq.value 1153
softirq.value 1728224

What do you get when you run that command manually?~~

Here is the result of my linode:

user.value 240019
nice.value 151044
system.value 85593
idle.value 188229895
iowait.value 156121
irq.value 0
softirq.value 1083

@mendel:

No, that graph isn't right, because Munin graphs idle time too.

I also see the idle time is not graphed properly by munin 1.2.6 (Ubuntu 8.10 default version) on a Linode with the most recent kernel version, see:

http://www.linode.com/forums/viewtopic. … n&start=15">http://www.linode.com/forums/viewtopic.php?t=3638&postdays=0&postorder=asc&highlight=munin&start=15

The most recent development version of munin apparently works properly.

James

Hi,

I'm using 2.6.28-linode15 kernel and munin 1.2.6.

Should I downgrade to a lower kernel version? wich one?

@nfn:

Should I downgrade to a lower kernel version?

The problem is in the Ubuntu default version of munin, and only affects the display of idle time - it's not colored most of the time. Everything else is just fine. No need to change kernel versions.

James

Hi James,

But how am I able to get this to work?

Thanks

Nuno

@nfn:

But how am I able to get this to work?

It works now - any part of the graph that is not colored is idle time.

James

This is still broken, the rrd file is showing NaN values for idle. I'm wondering if this is a Xen related bug.

not sure what you're talking about.

my munin graphs show up just fine with currently about 390 for idle (4 processors)

I'm running version 1.2.6 (Debian package 1.2.6-10) with kernel 2.6.29-linode18 and it's showing this:

![](http://linode.vlfsystems.com/munin/vlfs … pu-day.png">http://linode.vlfsystems.com/munin/vlfsystems.com/linode.vlfsystems.com-cpu-day.png" />

i'm running 1.2.6 as well but on kernel 2.6.18.8-linode16

the graph looks like this:

![](" />

Works fine for me; Ubuntu 8.04 LTS, munin-node 1.2.5-2ubuntu3.1, kernel 2.6.29-linode18. What do you get when you execute /etc/munin/plugins/cpu directly?

I get:

user.value 657225
nice.value 11049
system.value 751916
idle.value 1547600188
iowait.value 1161455
irq.value 23
softirq.value 12908

Also, what does the output of grep ^cpu /proc/stat look like? I get:

cpu  657303 11049 752022 1547746619 1161544 23 12909 46936 0
cpu0 193604 11017 254299 385028282 968977 22 11347 29859 0
cpu1 113176 24 168313 387648437 50962 0 156 6618 0
cpu2 265523 2 211470 387222893 89998 0 1020 6171 0
cpu3 84998 4 117939 387847005 51605 0 384 4288 0

Weird. Here is the output from the cpu plugin:

user.value 872302
nice.value 444187
system.value 382527
idle.value 556421712
iowait.value 4176544
irq.value 7
softirq.value 9615

And the grep command:

cpu  872310 444187 382537 556496028 4176681 7 9615 16509 0
cpu0 455215 140130 130080 135889890 3518150 7 4734 6676 0
cpu1 205119 115178 102059 139993137 259230 0 2092 4127 0
cpu2 119439 101066 82950 140224895 220081 0 1508 2975 0
cpu3 92535 87812 67446 140388105 179219 0 1280 2730 0

That looks right. Huh. Here's my /etc/munin/plugins/cpu :

http://pastebin.com/f34d43d8f

Maybe the rrd file is corrupt on the server end?

````
[ims@li19-162 ~]$ grep ^cpu /proc/stat
cpu 24092152 11250058 6593078 3165413444 27722384 0 68526 10211925
cpu0 15668438 2776785 2209170 761840941 25394297 0 43043 3405215
cpu1 2656302 2796856 1497932 801261831 872025 0 8743 2244200
cpu2 3019664 2820916 1461009 800988832 752089 0 8501 2286879
cpu3 2747747 2855499 1424965 801321839 703971 0 8238 2275629

and

user.value 24092153
nice.value 11250058
system.value 6593080
idle.value 3165416776
iowait.value 27722399
irq.value 0
softirq.value 68526

````

hoopycat got a good point, try re-initializing your rdd file(s)

Yeah I tried letting munin re-init the rrd files and that didn't fix it. The issue appears to be the "MAX" is set at 400 for the idle rrd file. This is based on the fact that there are 4 cpu's so the max value shouldn't be above 400. However, it looks like my data is in the 700's so the rrd get's NaN for everything. If I remove the maximum value I get data, but it doesn't make much sense. I still wonder if something is wrong with the kernel…

Oh and the version of cpu that was posted to pastebin is a little older and missing the maximum value setting.

Pastebin your copy of the cpu script? I'll give it a shot here…

Here you are, you'll notice the "MAX" setting on the rrd.

http://p.linode.com/2509

Indeed, that'd be the difference… aight, I have it installed here and will advise in awhile once it's had a chance to spin.

Yep, your script also breaks for me:

The CPU script on mine:

~~http://hoopycat.com/~rtucker/tmp/framboise-cpu-day.png" />

The CPU script from yours:

~~http://hoopycat.com/~rtucker/tmp/frambo … ew-day.png">http://hoopycat.com/~rtucker/tmp/framboise-cpu_new-day.png" />

Might want to open a bug report with the Munin folks… weird, that. (Secretly, I kinda like it without the idle, but that's just me ;-))~~~~