kjournald as source of disk io notification alert
i ran iotop -a -o -t -b | tee output.txt for a couple of hours, and the only thing out of the ordinary was kjournald. while i run a web server on this linode, there weren't any appreciable web requests to speak of.
even though kjournald was an outlier, it was only 60MB of disk io. i used vmstat simultaneously to try and line up the time of the disk io to the process that was causing it. the only thing i could find was that a spike in IO happened exactly 22 minutes apart. here are a few lines of vmstat output around the spike. the bo column always has upwards of 37000 blocks out, then another smaller line, and then yet smaller but noticeable couple of lines afterwards. you can also see the context switch column go wild.
are there tools i can use to determine the process that is running every 22 minutes in this way? is there a simple script i could write that monitors /proc in some way? i looked through user's crontabs and did not find anything in them. there are only a few users in the system (really just myself and some fake users for projects i am working on).
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 37156 33912 49288 325860 0 0 0 2 97 56 0 0 100 0
0 0 37156 33912 49288 325868 0 0 0 9 99 58 0 0 100 0
1 0 37156 30392 49300 327688 0 0 28 35730 16746 13233 3 3 82 12
0 0 37156 32664 49292 326312 0 0 1 22774 10248 8391 2 2 85 11
0 0 37156 32904 49292 326324 0 0 0 8 106 59 0 0 100 0
0 0 37156 33036 49292 326336 0 0 0 448 147 59 0 0 100 0
0 0 37156 33036 49292 326348 0 0 0 3 109 57 0 0 100 0
4 Replies
@shaheen:
i tried looking at all cron jobs for all users and didn't find any. are you suggesting there is another place i should be looking for cron jobs?
I was suggesting pseudo cron jobs. Cron jobs that are triggered by software and not by the actual cron tasks of the server. If you have for instance forum software on your server (say vBulletin, xenForo etc.) that has frequent visits per hour, some of the tasks that make that software run will be run from within the software and triggered by users/guests visits to that particular site/software.
I hope that helps.