One month in the life of host5

Mind if we ask how we can make such cool graphs ?

Bill Clinton (aka Sunny Dubey)

@Bill Clinton:

Mind if we ask how we can make such cool graphs ?

Bill Clinton (aka Sunny Dubey)

Well, there are probably really elegant ways to do it with MRTG/rrdtools, but I use a couple of simple scripts that I wrote instead because they were simpler. First I run the following script on my Linode:

#!/bin/sh

while true; do
    DATE=`date`
    echo -n "$DATE "
    uptime
    sleep 15
done

I run it like this:

./uptime.sh 2>&1 > uptime.out &

I let that run for as long as I want to; generally I have it running all of the time, and whenever I get around to it I convert the results into a graph with this script:

#!/bin/sh

cat uptime.out | awk '{ if($16=="average:") print $2,$3,$4,$6,$17,$18,$9; else print $2,$3,$4,$6,$16,$17,$18 }' | tr ',' ' ' > raw_data

echo 'set terminal jpeg; set yrange [0:20]; set xdata time; set bmargin 3; set timefmt "%b %d %H:%M:%S %Y"; plot "raw_data" using 1:5 title "1 min load average" with lines' | gnuplot > plot.jpg

display plot.jpg

That's it. Pretty simple really.

It relies on 'uptime' to determine the current host load; in general because my Linode is idle almost all of the time, any significant load is due to other Linodes hogging all of the disk I/O. When that happens, processes sit on the kernel's run queue while they wait for I/O requests to complete, and when that happens, the load level goes up. Thus we get a pretty good correlation between the disk activity on the host system and any particular Linode's load level.

For what it's worth, the spikes are pretty short lived even though they are annoying. Caker offered to move me to another host but it's such a minor issue that I don't even want to bother. I'd like to see a UML kernel fix for this problem though - disk I/O should be shared as fairly as CPU! Maybe someday someone will enhance Linux to handle this better …

@bji:

echo 'set terminal jpeg; set yrange [0:20]; set xdata time; set bmargin 3; set timefmt "%b %d %H:%M:%S %Y"; plot "raw_data" using 1:5 title "1 min load average" with lines' | gnuplot > plot.jpg

I thought that looked like a gnuplot graph :-)

@bji:

I'd like to see a UML kernel fix for this problem though - disk I/O should be shared as fairly as CPU! Maybe someday someone will enhance Linux to handle this better …
The problem is, random I/O always performs the worst out of any I/O pattern. Multiply that by X number of threads (linodes), and bandwidth decreases severely. I removed the slocate cron jobs from the template distros earlier this year, so new deployments (and hosts) don't exhibit this problem as severely as some of the original hosts. The only solution, other than throttling, is to have those cron jobs removed. When those hosts are rebooted into 2.6, I will be writing a script to remove them before bringing the Linodes back up.

The token-limiter patch (as outlined in this thread, source here) provides me with a mechanism to throttle I/O coming from each individual UML. The patch by itself protects the host from being over-taken by a single Linode using all of the available I/O bandwidth (for example, by swap thrashing), but with other settings I can change the behavior…

The next step will be to design a "monitor" for the host which dynamically changes the limiter's values during runtime, depending on host-load (/me pats /me on back). I can and will make it sensitive enough to throttle I/O during cron parties. I'll set something simple up this evening to fire off before cron-time, let me know if you notice any improvement.

In effect, this patch can push the load back onto the Linode that's consuming I/O, rather than onto the host and the other Linodes.

-Chris

How can I tell how much I/O bandwidth my linode is burning? Right now, for example, my linode is sluggish as hell – load over 3.5 -- yet my CPU is idle.

PS: yes, I've changed the time of my updatedb for locate.

Well there are 2 ways.

The first is via list where you can use the io_status command

The other is if you are using the latest kernel deployed yesterday you can find it it /proc/io_status

Adam

Thanks man, I shall try later today

I would just like to point out something (if you don't mind). I've noticed that your shell code involves

#!/bin/sh

while true; do
    [...]
done

You should replace the "true" with a single colon. Essentially this prevents bash from creating a mindless subshell to yield a true value

#!/bin/sh

while : ; do
    [...]
done

@Bill Clinton:

You should replace the "true" with a single colon. Essentially this prevents bash from creating a mindless subshell to yield a true value

Isn't true a bash builtin?

This is my very first post – I'll try not to screw it up too bad…

So far, I'm crazy about Linode…

If anyone's interested, this is a fairly efficient way of maintaining a load average history file -- it's fast and the history file is relatively small. No need to run in background, as this is done by the 'daemon' command.

Although the slashes in the date take up more file space, they make it much eaiser to search for a range using the grep command.

Of course, there are many changes and improvements that could be made…

Hal Williams

/*
loadavg.c
to compile: cc -s -o loadavg loadavg.c
usage: loadavg [history_file_name]
*/

#include <stdio.h>#include <time.h>#include <unistd.h>/* if not specified on command line */
#define DEF_LOG "/etc/loadavg.log"

static time_t t;
static double avg[3];
static struct tm ts;
static FILE *file;

main(int argc, char *argv[])
{

/* create if file doesn't exist */
/* always attempt to append to end of file */
if (! (file = fopen((argc > 1) ? argv[1] : DEF_LOG,"a")))
   exit(1);

if (daemon(0,0) < 0)
   exit(1);

setbuf(file,NULL);            /* probably not needed */

while(1)
   {
   time(&t);
   if (getloadavg(avg,3) != 3)
       exit(1);
   if (! localtime_r(&t,&ts))
       exit(1);
   fprintf(file,"%02d/%02d/%02d %2d:%02d %3.1f %3.1f %3.1f\n",
       ts.tm_mon+1,ts.tm_mday,ts.tm_year%100,
       ts.tm_hour,ts.tm_min,
       avg[0],avg[1],avg[2]);
       sleep(60);
   }
}</unistd.h></time.h></stdio.h> 

@hwilliams:

t's fast and the history file is relatively small. No need to run in background, as this is done by the 'daemon' command.

Although the slashes in the date take up more file space, they make it much eaiser to search for a range using the grep command.
If you're worried about space, make this program write the data out in binary (that'd work out around 7 or 8 bytes per minute) and then have a small program which reads this binary format and displays it in text readable form (which can then be passed to grep etc).

Essentially this is what programs like "sar" (in the "sysstat" package) do; a program writes binary data and the "sar" command just parses that whenever the user wants to see the historical record.

@jeffml:

@Bill Clinton:

You should replace the "true" with a single colon. Essentially this prevents bash from creating a mindless subshell to yield a true value

Isn't true a bash builtin?

annie:~$ type true
true is a shell builtin
annie:~$ 

Yes.