kjournald as source of disk io notification alert

i've recently been getting lots of disk io rate notifications in email. i looked at the 24 hour graph and found a cyclical pattern. i then looked at the 30d graph and found a scarier thing: the average io rate has been trending upward linearly the entire month.

i ran iotop -a -o -t -b | tee output.txt for a couple of hours, and the only thing out of the ordinary was kjournald. while i run a web server on this linode, there weren't any appreciable web requests to speak of.

even though kjournald was an outlier, it was only 60MB of disk io. i used vmstat simultaneously to try and line up the time of the disk io to the process that was causing it. the only thing i could find was that a spike in IO happened exactly 22 minutes apart. here are a few lines of vmstat output around the spike. the bo column always has upwards of 37000 blocks out, then another smaller line, and then yet smaller but noticeable couple of lines afterwards. you can also see the context switch column go wild.

are there tools i can use to determine the process that is running every 22 minutes in this way? is there a simple script i could write that monitors /proc in some way? i looked through user's crontabs and did not find anything in them. there are only a few users in the system (really just myself and some fake users for projects i am working on).

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
0  0  37156  33912  49288 325860    0    0     0     2   97   56  0  0 100  0
 0  0  37156  33912  49288 325868    0    0     0     9   99   58  0  0 100  0
 1  0  37156  30392  49300 327688    0    0    28 35730 16746 13233  3  3 82 12
 0  0  37156  32664  49292 326312    0    0     1 22774 10248 8391  2  2 85 11
 0  0  37156  32904  49292 326324    0    0     0     8  106   59  0  0 100  0
 0  0  37156  33036  49292 326336    0    0     0   448  147   59  0  0 100  0
 0  0  37156  33036  49292 326348    0    0     0     3  109   57  0  0 100  0

4 Replies

Just a thought: How about cron jobs from software? For instance if you are running a vBulletin forum, they implement their own cron tasks based on traffic. 22 minutes could very well be that or something similar.

i tried looking at all cron jobs for all users and didn't find any. are you suggesting there is another place i should be looking for cron jobs?

@shaheen:

i tried looking at all cron jobs for all users and didn't find any. are you suggesting there is another place i should be looking for cron jobs?

I was suggesting pseudo cron jobs. Cron jobs that are triggered by software and not by the actual cron tasks of the server. If you have for instance forum software on your server (say vBulletin, xenForo etc.) that has frequent visits per hour, some of the tasks that make that software run will be run from within the software and triggered by users/guests visits to that particular site/software.

I hope that helps.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct