Unusual CPU usage on dallas58/UML

How does your CPU graph on the platform manager look?

Also, NPTL libc isn't a problem with Xen on a 64-bit host (as linode uses), or on UML at all.

Hello Martin,

There are no incompatibilities or issues with NPTL in either our Xen or UML setups. UML has supported NPTL for quite some time. Also, our Xen config doesn't even require a modified libc to run at full speed (the hypervisor runs in 64bits).

I can verify that your Linode is spinning CPU on the host.

-Chris

Thoguht I'd give my 5 cents, I had unusual high loads as well recently, and the solution was that mysqld_safe was running twice.

Stop mysql, use killall mysqld_safe and the nstart mysql again and see if that helps.

@bdonlan:

How does your CPU graph on the platform manager look?
It looks something like this:

~~http://www.ultimalinux.com/~multima/cpu-usage.png" />

By the way, it's running kernel 2.6.23.17-linode43 (although I don't think there were any other choices).

@bdonlan:

Also, NPTL libc isn't a problem with Xen on a 64-bit host (as linode uses), or on UML at all.
I'll have to keep that in mind - I've never been good with libc-related stuff, I'm happy to do whatever so long as it works :)~~

@Marius:

Thoguht I'd give my 5 cents, I had unusual high loads as well recently, and the solution was that mysqld_safe was running twice.

Stop mysql, use killall mysqldsafe and the nstart mysql again and see if that helps. I have MySQL completely stopped, and it's still giving 80% sys in top. While it was running there was only one mysqld and one mysqldsafe process.

@Caker:

There are no incompatibilities or issues with NPTL in either our Xen or UML setups. UML has supported NPTL for quite some time. Also, our Xen config doesn't even require a modified libc to run at full speed (the hypervisor runs in 64bits).
Well, I certainly couldn't hope for anything more definitive than that.

@Caker:

I can verify that your Linode is spinning CPU on the host.
As long as you're aware, that's my main concern - I actually came back to Linode after my previous hosting was compromised and subsequently deleted. (Learned my lesson about third-party shared hosting, and settling for anything less than SSH…) I certainly wouldn't want my new site getting removed after all that :)

Edit I did a bit of experimentation, and the CPU usage appears to have subsided after shutting down inetd. I have a feeling it might have been some program acting up because I was re-configuring the Linode's hostname. Now I have 0% user, 0-5% sys, but 100% id (offhand I have no idea what that even is).

Edit Oops… sorry for double-posting, thought someone else had already replied again :oops:

id is idle.

from man top:

> us – User CPU time

The time the CPU has spent running usersâ processes that are not niced.

sy -- System CPU time

The time the CPU has spent running the kernel and its processes.

ni -- Nice CPU time

The time the CPU has spent running users proccess that have been niced.

wa -- iowait

Amount of time the CPU has been waiting for I/O to complete.

hi -- Hardware IRQ

The amount of time the CPU has been servicing hardware interrupts.

si -- Software Interrupts

The amount of time the CPU has been servicing software interrupts.

st -- Steal Time

The amount of CPU stolen from this virtual machine by the hypervisor for other tasks (such as running

another virtual machine).

@multima:

The only thing I can think of is that the installation uses an NPTL-only glibc, which I believe is how newer Slackware releases are also built (my code's originally forked from a 2006-ish Slackware-current, and has since been independently developed). I saw that's a bit of a problem with Xen, could that be impacting UML as well?

Slackware doesn't work in Xen? My UML and Xen Linodes both beg to differ.

@OverlordQ:

id is idle.

from man top:

…quote from man page here…
Another thing I'll have to remember… :roll: You can probably tell I haven't used top much before (I tend to prefer a combination of ps -A, killall -9, and free -m, and for everything else I just eyeball it…)

I'm assuming that 'idle' is what we're going for, or…? On my machines here, I've dabbled a bit, but never seriously monitored CPU utilization much - they're either idling, or else running builds, usually two or three simultaneously, at which point starting the CPU monitor frequently crashes the machine.

@mwalling:

Slackware doesn't work in Xen? My UML and Xen Linodes both beg to differ.
I don't think I meant it that way, but I will freely admit I have little UML-specific experience, and none whatsoever with Xen.

At any rate, CPU usage appears to be much better now:

~~http://www.ultimalinux.com/~multima/cpu-usage-2.png" />

I'll test it on one of my machines here, just to be sure, but I think it was some errant inetd process that didn't like my hostname changes, judging from (a) the CPU usage calming after restarting inetd, and (b) the fact that I saw a few "inetd, 100% usage" processes flash on my screen…

If everything checks OK, I guess I'll consider this one solved - my Linode's running fine now otherwise, but I may be back…~~

inetd may have started a child that was burning your CPU.

Next time, try "ps faux" if your system has that. It gives you a tree of processes with memory and CPU usage. (It helps to have a wide terminal for this.)

@dseagrav:

inetd may have started a child that was burning your CPU.
That's what I suspected. I reached that particular conclusion independently, about three days ago. :)

@dseagrav:

Next time, try "ps faux" if your system has that. It gives you a tree of processes with memory and CPU usage. (It helps to have a wide terminal for this.)
I'm aware of the ps command; I'm not particularly big into the -f option, or -aux, or anything that involves horizontally resizing my terminal window. (I generally have two or three running at any given moment, and even dual 1280x1024 displays only have so much screen space available.) I prefer a simple ps -A myself, since it's less cluttered, and it's easier to tell at a glance what should/shouldn't be running. But thanks for the tip.

I've currently kludged it by setting a cron job to restart inetd every morning around 4 am; this has worked nicely enough the last several days. I'm planning to investigate the cause of the problem once I have more time, possibly this weekend, but meanwhile I'm still working to rebuild the Web site - I had a copy of its database, but it was deleted before I could get a good backup of the files - and not to mention, there's also that whole issue of developing the actual product…

UPDATE: I should have posted this almost a month ago, but I've been a bit busy. Oh well.

Anyway, a glance through /var/log/secure later, it turns out that inetd was trying to spawn an in.identd process, but the executable in question wasn't on the system. (I forgot so many services were enabled by default…) I have since disabled that particular service, as well as several others that I don't use, and the CPU usage problems have not come up since.

I also discovered in syslog a rather absurd number of attempts on the SSH port - I guess just like everyone else here, judging from all the other topics ;) That was around April 13. My new sshd_config has put an end to all that, although I don't think it's quite draconian enough yet, even considering - 15 minutes grace time is still a lot, and I should probably disable interactive password entry since I always use SSH keys (and I'm the only current user on the site).