High disk IO - Is it normal?

Hi,

Something is forcing your linode into quite extreme disk usage. This can be for a number of reasons, all because of a quite unhealthy setup.

One of the main things to consider is whether or not your linode is doing extreme amounts of swapping.

Check what amount of memory you have free with the command "free".

It should give an output similar to this:

             total       used       free     shared    buffers     cached
Mem:       2047728      64104    1983624          0       2164      21248
-/+ buffers/cache:      40692    2007036
Swap:       524284          0     524284

As you can see, I'm using almost no memory, and no swap at all.

Checking this is a good place to start - swapping isn't healthy in this sort of environment.

And, even if this isn't the cause, there is something fundamentally wrong with your setup - it should never have this high I/O, and even the smallest linode should handle the job you require x10 with ease.

Also try installing iotop it'll tell you what's using your io

Thanks for helping.

This what free outputs:

             total       used       free     shared    buffers     cached
Mem:        510652     299760     210892          0        652      10676
-/+ buffers/cache:     288432     222220
Swap:       262136      55600     206536

I installed iotop;

And this worried me during install:

"Processing triggers for man-db …

/usr/bin/mandb: can't write to /var/cache/man/8588: No space left on device"

So I checked space left: 0%

I was, to say the least, puzzled. I am supposed to have 16Gigs, of which a good chunk is taken by buntu, but all the rest is just a few php/html/js files and a few low-res images.

I quickly cleaned apt cache because the system was being extremely unresponsive.

so i tried to sniff around; my apache log files in var/log/apache2 were takin up 4gigs. Is that normal?

my /home/ is on another drive so that's not it.

So now my problem is: What is takin up my space?

4GB of logs is quite a bit. Is that in a single log that you're not rotating? I would suggest putting in logrotate if you don't have it already.

du -m / | sort -n | head

And well, huge apache logs may happen, especially if you become a target of a botnet.

That's one of the reasons why keeping /var/log on a separate partition may be a good idea - to prevent those from filling the whole disk.

And yes, my Linode suffered from that too, tho it wasn't my code update which caused the errorlog to GROW…

Ok, this is the output from du:

du: cannot access `/proc/20018/task/20020/fd/28': No such file or directory
du: cannot access `/proc/20359/task/20359/fdinfo/71': No such file or directory
du: cannot access `/proc/20411/task/20411/fd/163': No such file or directory
du: cannot access `/proc/20411/task/20414/fd/30': No such file or directory
du: cannot access `/proc/20456/task/20468/fd/148': No such file or directory
du: cannot access `/proc/20457/fdinfo/138': No such file or directory
du: cannot access `/proc/20476/task/20476/fd/4': No such file or directory
du: cannot access `/proc/20476/task/20476/fdinfo/4': No such file or directory
du: cannot access `/proc/20476/fd/4': No such file or directory
du: cannot access `/proc/20476/fdinfo/4': No such file or directory
0    /dev/block
0    /dev/char
0    /dev/disk
0    /dev/disk/by-path
0    /dev/disk/by-uuid
0    /dev/input
0    /dev/mapper
0    /dev/net
0    /dev/pts
0    /dev/shm

I have the feeling this is not normal.

I am thinking of just wiping my partition clean and do a fresh install of ubuntu.

[edit] but then I would be ever frustrated because I would not have solved the mystery

That's normal.

Those processes probably exited between du getting a directory listing and being able to process it.

The /dev subsystem is special, those generally dont take up hard-drive space they're just mappings.

Use du -m / | sort -n -r | head to sort by descending size.

Try sudo du -m --max-depth 1 / | sort -n

@obs:

Use du -m / | sort -n -r | head to sort by descending size.
Whoops >.>; sorry… (well, | tail witohut reverse sort would work too… either way, I screwed up :P)