Linode Alert CPU usage - what happened?
> Your Linode has exceeded the notification threshold (90) for CPU Usage by averaging 124.6% for the last 2 hours.
I received Linode Alerts before, but that was while I was doing performance tests. Today I wasn't. How do I figure out what happened?
My munin installation is incomplete (just default), because the munin site gives 500 errors when trying to download a plugin for almost 2 weeks now.
What do I see on munin graphs:
munin no activity between 6 and 9 am
a peak at connections through firewall just before 6 am
a peak at postfix bytes throughput at half past 6
fork rate, cpu usage, and interrupts peak between 6.30 and 9.00 am
gaps in the charts for number of threads, process priority, vmstat, file table usage, memory usage around that time.
Don't think it is an outside job (DOS, nothing in awstats or log files looks unusual). I have created 2 custom cronjobs yesterday, but they run between 23.00 and 23.10. Around half past 6 is the time when logrotate is scheduled and logwatch sends its report around that time too.
I noticed that I am sending a lot of messages to myself (mail.log)
Jan 10 06:40:05 m41l postfix/local[29928]: 4D434B34F: to=<www-data@m41l.example.com>, orig_to=<www-data>, relay=local, delay=0.03, delays=0.02/0.01/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jan 10 06:40:05 m41l postfix/qmgr[2377]: 4D434B34F: removed
Jan 10 06:50:05 m41l postfix/pickup[4029]: 17379B34F: uid=33 from= <www-data>Jan 10 06:50:05 m41l postfix/cleanup[24869]: 17379B34F: message-id=<20120110055005.17379B34F@m41l.example.com>
Jan 10 06:50:05 t4d0rn4 postfix/qmgr[2377]: 17379B34F: from=<www-data@m41l.example.com>, size=886, nrcpt=1 (queue active)</www-data@m41l.example.com></www-data></www-data></www-data@m41l.example.com>
Where m41l.example.com is the hostname of my server (modified). Half past 6 it seemed like there were a 1000 mails in queue. Don't know what for. Don't know where the mails for
Where do I start looking? How do I check if there is any mail for root or www-data? Or how do I divert it to another email address? How do I check which programs are trying to send me email? Logwatch and custom cronjobs (using php mailer) work fine in sending me messages (to an outside email address).
Anyway, will see what happens tomorrow, bit puzzled right now.
Still don't know what caused the surge.
In Debian-based distributions (including Ubuntu), Postfix stores local mail in /var/mail by default. I'm not sure about other distros, but I suspect it's the same.
Most of domains on server are parked without contact email, maybe I should create catch-all email addresses per domain (hope I can do this with Google Apps).
Had a look in /var/mail, it was very enlightening.
The mails to root are munin cronjobs that failed.
The mails to www-data are awstats cronjobs that reported an error.
Will have a look at them tomorrow.
