What is kernel panic?

"Kernel panic" is the Linux equivalent of a Windows "Blue Screen of Death" - meaning, an unrecoverable system error.

http://en.wikipedia.org/wiki/Kernel_panic

When it happens, the system will have to be restarted (or in the case of virtual machines, your particular instance has to be rebooted).

If you log into lish you should see a panic message on the console screen. Otherwise check your /var/log/syslog or /var/log/messages for kernel related issues (it'll say kernel next to the message).

It could just be a case you're running out of ram and your server is locking up. Can provide more details about the restarts i.e. do they happen automatically or are you doing it manually?

It happens automatically, but my ram only consumes 40% I'm using plan 768. How can I stop this?

You probably want it to reboot automatically on a kernel panic (if it is a kernel panic) otherwise the server will just stop working.

Log into lish, type logview and paste the contents here. Then we will see if it is a kernel panic.

I typed in "logview" but it seems its not the right command? I'm sorry I still new to Linux regarding this problems.

Have a read of this http://library.linode.com/troubleshooti … node-shell">http://library.linode.com/troubleshooting/using-lish-the-linode-shell it tells you how to use lish including the logview command

When I try to login now via LISH, my screen went crazy.

Are you using lish via ssh or your browser? If via your browser use ssh it's better. If not define "crazy"

It pops out too many words. But after few minutes, the screen froze stating that Console starting… but I can't type anything.. .Should I wait on this to boot?

I'm using PuTTy.

Try pressing ctrl+a then keep ctrl held down and press d

If that doesn't work raise a ticket with linode support to see if there's something wrong with your lish.

Alright, here's what I got from logview

top - 18:23:45 up 26 min,  1 user,  load average: 0.09, 0.04, 0.05
Tasks:  99 total,   1 running,  98 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    768512k total,   480788k used,   287724k free,    10716k buffers
Swap:   524284k total,        0k used,   524284k free,   152604k cached

Broadcast message from root (Fri Nov 25 18:23:47 2011):IME+  COMMAND
 6137 root      20   0  264m 242m 2716 S  0.7 32.3   0:06.38 map-server_sql
The system is going down for system halt NOW!  0.1   0:00.14 top
INIT: Switching to runlevel: 0616  528 S  0.0  0.1   0:00.46 init
INIT: Sending processes the TERM signalS  0.0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.03 kworker/u:0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/1:0
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1
   10 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2
   11 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/2:0
   12 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2
   13 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3
   14 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/3:0
   15 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3
Stopping yum-updatesd: [  OK  ]
Stopping atd: [  OK  ]
Shutting down console mouse services: [  OK  ]
Stopping httpd: [  OK  ]
Stopping sshd: [  OK  ]
Shutting down sm-client: [  OK  ]
Shutting down sendmail: [  OK  ]
Stopping MySQL:  [  OK  ]
Stopping crond: [  OK  ]
Stopping autofs:  Stopping automount: [  OK  ]
[  OK  ]
Shutting down ntpd: [  OK  ]
Stopping system message bus: [  OK  ]
Stopping auditd: type=1305 audit(1322216636.690:29): audit_pid=0 old=2968 auid=4                                                                                       294967295 ses=4294967295 res=1
[  OK  ]
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
find: /sys/class/iscsi_session/: No such file or directory
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Flushing firewall rules: type=1325 audit(1322216639.046:30): table=security fami                                                                                       ly=2 entries=4
type=1300 audit(1322216639.046:30): arch=40000003 syscall=102 success=yes exit=0                                                                                        a0=e a1=bfe036e0 a2=8055884 a3=8e186a4 items=0 ppid=7545 pid=7554 auid=42949672                                                                                       95 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=429496                                                                                       7295 comm="iptables" exe="/sbin/iptables" key=(null)
type=1325 audit(1322216639.051:31): table=security family=2 entries=4
type=1300 audit(1322216639.051:31): arch=40000003 syscall=102 success=yes exit=0                                                                                        a0=e a1=bf98e560 a2=8055884 a3=9f886a4 items=0 ppid=7545 pid=7556 auid=42949672                                                                                       95 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=429496                                                                                       7295 comm="iptables" exe="/sbin/iptables" key=(null)
type=1325 audit(1322216639.053:32): table=raw family=2 entries=3
type=1300 audit(1322216639.053:32): arch=40000003 syscall=102 success=yes exit=0                                                                                        a0=e a1=bf84ae60 a2=8055884 a3=90ea508 items=0 ppid=7545 pid=7557 auid=42949672                                                                                       95 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=429496                                                                                       7295 comm="iptables" exe="/sbin/iptables" key=(null)
type=1325 audit(1322216639.057:33): table=raw family=2 entries=3
type=1300 audit(1322216639.057:33): arch=40000003 syscall=102 success=yes exit=0                                                                                        a0=e a1=bfd3fab0 a2=8055884 a3=95e6508 items=0 ppid=7545 pid=7559 auid=42949672                                                                                       95 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=429496                                                                                       7295 comm="iptables" exe="/sbin/iptables" key=(null)
type=1325 audit(1322216639.060:34): table=nat family=2 entries=5
[  OK  ]
Setting chains to policy ACCEPT: security raw nat mangle filter [FAILED]
Unloading iptables modules: [  OK  ]
Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
 enable [-pnds] [-a] [-f filename]  eval [arg ...]
 xec [-cl] [-a name] file [redirec exit [n]
 xport [-nf] [name[=value] ...] or false
 c [-e ename] [-nlr] [first] [last fg [job_spec]
 or NAME [in WORDS ... ;] do COMMA for (( exp1; exp2; exp3 )); do COM
 unction NAME { COMMANDS ; } or NA getopts optstring name [arg]
 ash [-lr] [-p pathname] [-dt] [na help [-s] [pattern ...]
history [-c] [-d offset] [n] or hi if COMMANDS; then COMMANDS; [ elif
Syncing hardware clock to system time audit_printk_skb: 111 callbacks suppressed
 ype=1111 audit(1322216645.953:53): local name[=value] ...id=4294967295 ses=4294                                                                                        ogout msg='changing system time: expopd [+N | -N] [-n]hostname=?, addr=?, termi                                                                                        rintf [-v var] format [arguments] pushd [dir | +N | -N] [-n]
 wd [-LP]                           read [-ers] [-u fd] [-t timeout] [
 eadonly [-af] [name[=value] ...]  return [n]
 elect NAME [in WORDS ... ;] do CO set [--abefhkmnptuvxBCHP] [-o opti
 hift [n]lted.                      shopt [-pqsu] [-o long-option] opt
 ource filename [arguments]         suspend [-f]
 est [expr]t 100 lines from current time [-p] PIPELINE
 imes-------------------------------trap [-lp] [arg signal_spec ...]
 ruetion NAME { COMMANDS ; } or NA gtype [-afptP] name [name ...]
 ypeset [-afFirtx] [-p] name[=valu ulimit [-SHacdfilmnpqstuvx] [limit
 mask [-p] [-S] [mode]t] [n] or hi iunalias [-a] name [name ...] elif
 nset [-f] [-v] [name ...]] or job kuntil COMMANDS; do COMMANDS; done
variables - Some variable names an wait [n]
 enable [-pnds] [-a] [-f filename]  eval [arg ...]
 xec [-cl] [-a name] file [redirec exit [n]
 xport [-nf] [name[=value] ...] or false
 c [-e ename] [-nlr] [first] [last fg [job_spec]
 or NAME [in WORDS ... ;] do COMMA for (( exp1; exp2; exp3 )); do COM
 unction NAME { COMMANDS ; } or NA getopts optstring name [arg]
 ash [-lr] [-p pathname] [-dt] [na help [-s] [pattern ...]
 istory [-c] [-d offset] [n] or hi if COMMANDS; then COMMANDS; [ elif
 enable [-pnds] [-a] [-f filename]  eval [arg ...]
 xec [-cl] [-a name] file [redirec exit [n]
 xport [-nf] [name[=value] ...] or false
 c [-e ename] [-nlr] [first] [last fg [job_spec]
 or NAME [in WORDS ... ;] do COMMA for (( exp1; exp2; exp3 )); do COM
 unction NAME { COMMANDS ; } or NA getopts optstring name [arg]
 ash [-lr] [-p pathname] [-dt] [na help [-s] [pattern ...]
history [-c] [-d offset] [n] or hi if COMMANDS; then COMMANDS; [ elif
 obs [-lnprs] [jobspec ...] or job kill [-s sigspec | -n signum | -si
 et arg [arg ...]                   local name[=value] ...
 ogout                              popd [+N | -N] [-n]
 rintf [-v var] format [arguments] pushd [dir | +N | -N] [-n]
 wd [-LP]                           read [-ers] [-u fd] [-t timeout] [
 eadonly [-af] [name[=value] ...]  return [n]
 elect NAME [in WORDS ... ;] do CO set [--abefhkmnptuvxBCHP] [-o opti
 hift [n]                           shopt [-pqsu] [-o long-option] opt
 ource filename [arguments]         suspend [-f]
 est [expr]                         time [-p] PIPELINE
 imes                               trap [-lp] [arg signal_spec ...]
 rue                                type [-afptP] name [name ...]
 ypeset [-afFirtx] [-p] name[=valu ulimit [-SHacdfilmnpqstuvx] [limit
 mask [-p] [-S] [mode]              unalias [-a] name [name ...]
 nset [-f] [-v] [name ...]          until COMMANDS; do COMMANDS; done
variables - Some variable names an wait [n]
 enable [-pnds] [-a] [-f filename]  eval [arg ...]
 xec [-cl] [-a name] file [redirec exit [n]
 xport [-nf] [name[=value] ...] or false
 c [-e ename] [-nlr] [first] [last fg [job_spec]
 or NAME [in WORDS ... ;] do COMMA for (( exp1; exp2; exp3 )); do COM
 unction NAME { COMMANDS ; } or NA getopts optstring name [arg]
 ash [-lr] [-p pathname] [-dt] [na help [-s] [pattern ...]
 istory [-c] [-d offset] [n] or hi if COMMANDS; then COMMANDS; [ elif
 obs [-lnprs] [jobspec ...] or job kill [-s sigspec | -n signum | -si
 et arg [arg ...]                   local name[=value] ...
 ogout                              popd [+N | -N] [-n]
 rintf [-v var] format [arguments] pushd [dir | +N | -N] [-n]
 wd [-LP]                           read [-ers] [-u fd] [-t timeout] [
 eadonly [-af] [name[=value] ...]  return [n]
 elect NAME [in WORDS ... ;] do CO set [--abefhkmnptuvxBCHP] [-o opti
 hift [n]                           shopt [-pqsu] [-o long-option] opt
 ource filename [arguments]         suspend [-f]
 est [expr]                         time [-p] PIPELINE
 imes                               trap [-lp] [arg signal_spec ...]
 rue                                type [-afptP] name [name ...]
 ypeset [-afFirtx] [-p] name[=valu ulimit [-SHacdfilmnpqstuvx] [limit
 mask [-p] [-S] [mode]              unalias [-a] name [name ...]
 nset [-f] [-v] [name ...]          until COMMANDS; do COMMANDS; done
variables - Some variable names an wait [n]
 while COMMANDS; do COMMANDS; done  { COMMANDS ; 

It's not a kernel panic, something sent your server the shutdown command.

Broadcast message from root (Fri Nov 25 18:23:47 2011)

The system is going down for system halt NOW

So either a cron job, someone logged in via ssh or some other root process sent the signal.

It's impossible that someone know my linode, but cron job is something I should look, and another noob question. How do I know what are cron jobs in my linode?

It's never impossible that someone else has access to your linode.

For crons run crontab -l as root also check all files in the /etc/cron.* folders.

What does @reboot mean on my crontab?

Anyway, to keep myself at ease.. How can I change my linode's password? Meaning the machine itself.

@reboot means it runs when your server boots.

To change your root password when logged in as root run passwd however if someone else already has root access then changing your password won't help since there's other ways they can get in.

Is there any IP log to check if someone's accessing?

Yes and no, logins are logged in /var/log/auth.log (assuming your distribution logs them there it might not) you can also check /var/log/syslog

However anyone with root access can edit these logs.

Oh and btw, what do you mean they can still access it even if I changed the password?

Say someone created a ssh key for the root user they could bypass the password, there's loads of other ways too. Basically if someone gets root access you're stuffed.

That's why you need to go find out if it's just some bad configuration and you've a scheduled reboot command lurking somewhere or if it's something malicious.

How do I know if I had a scheduled reboot, and how do I know if they created an ssh key on my linode?

Scheduled reboot would be something like a cron or at job (have a google for cron job ), and I wouldn't worry about checking if someone created a ssh key like I said if they gained root access you're stuffed you'll have to rebuild your server.

Thank you for your time! If someone (hopefully none) created an ssh via root, if I rebuild the OS does it mean it will gone?

Yep since the server will be wiped, however if someone did compromise your server you'll need to make sure they can't do it again following good practices, i.e. only use ssh keys for shell access, run external processes as restricted users i.e. php/python/ruby, configure iptables firewall, keep your software up to date etc

Thank you so much for giving me your time. But regarding using ssh keys for shell access only, how can I do that? Does shell access means using a software like Putty?

Have a read of this for how to use ssh keys http://library.linode.com/security/ssh-keys

Once you have working ssh keys you'll want to edit /etc/ssh/sshd_config and set PasswordAuthentication no

What does this log mean? It restarted again although I haven't rebuild the OS yet.

Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D     3.0.4-linode38 #1
Call Trace:
 [<c06bd3b6>] ? panic+0x57/0x13e
 [<c011f160>] ? mm_fault_error+0x130/0x130
 [<c010b7f6>] ? oops_end+0x96/0xa0
 [<c011ef7f>] ? bad_area_nosemaphore+0xf/0x20
 [<c011f3af>] ? do_page_fault+0x24f/0x3a0
 [<c04570c7>] ? cpumask_next_and+0x17/0x30
 [<c0130d18>] ? find_busiest_group+0x138/0xa40
 [<c018b7a4>] ? __alloc_pages_nodemask+0xf4/0x630
 [<c011f160>] ? mm_fault_error+0x130/0x130
 [<c06c0106>] ? error_code+0x5a/0x60
 [<c011f160>] ? mm_fault_error+0x130/0x130
 [<c05c02f0>] ? ipv4_dst_check+0x90/0x150
 [<c0536a88>] ? __sk_dst_check+0x38/0x60
 [<c05c7a91>] ? ip_queue_xmit+0x251/0x3d0
 [<c05daf25>] ? tcp_transmit_skb+0x385/0x670
 [<c05db379>] ? tcp_send_ack+0xb9/0x100
 [<c05deb11>] ? tcp_delack_timer+0x101/0x1d0
 [<c013fbb3>] ? run_timer_softirq+0xe3/0x1f0
 [<c0131c19>] ? rebalance_domains+0x79/0x140
 [<c05dea10>] ? tcp_keepalive_timer+0x230/0x230
 [<c0139cfc>] ? __do_softirq+0x7c/0x110
 [<c0139c80>] ? irq_enter+0x60/0x60
 <irq>  [<c0139ade>] ? irq_exit+0x6e/0xa0
 [<c047829d>] ? xen_evtchn_do_upcall+0x1d/0x30
 [<c06c0947>] ? xen_do_upcall+0x7/0xc
 [<c01013a7>] ? hypercall_page+0x3a7/0x1000
 [<c0105c7f>] ? xen_safe_halt+0xf/0x20
 [<c010f41e>] ? default_idle+0x2e/0x60
 [<c0107f72>] ? cpu_idle+0x42/0x70</c0107f72></c010f41e></c0105c7f></c01013a7></c06c0947></c047829d></c0139ade></irq></c0139c80></c0139cfc></c05dea10></c0131c19></c013fbb3></c05deb11></c05db379></c05daf25></c05c7a91></c0536a88></c05c02f0></c011f160></c06c0106></c011f160></c018b7a4></c0130d18></c04570c7></c011f3af></c011ef7f></c010b7f6></c011f160></c06bd3b6>

Ironically that is a kernel panic, looks like one similar to what other people get with 3.0.4 see http://forum.linode.com/viewtopic.php?t=7805

What kernel should I use? I'm currently using kernel 3.0.4

3.0.4 is the latest "stable" I've a couple of servers than run it without issues, others run the distros stock kernels using pv_grub. You could poke linode support with the panic message and see if they have a way of fixing it yet.

The other thread isn't a kernel panic. The first line of the call trace says "warnallocfailed" if it's nothing to worry about, but "panic" if it is. This says "panic".

There's normally a lot more after this in the dump, particularly a Mem-Info section. That will determine, at a glance, if it's an actual bug or if you're just running out of memory. (Free swap = 0kB means you ran out of memory.)

What should I do to prevent this panic?

@hoopycat:

The other thread isn't a kernel panic. The first line of the call trace says "warnallocfailed" if it's nothing to worry about, but "panic" if it is. This says "panic".
Oops…must learn to read posts fully

@hoopycat:

The other thread isn't a kernel panic. The first line of the call trace says "warnallocfailed" if it's nothing to worry about, but "panic" if it is. This says "panic".

There's normally a lot more after this in the dump, particularly a Mem-Info section. That will determine, at a glance, if it's an actual bug or if you're just running out of memory. (Free swap = 0kB means you ran out of memory.)

How can I eliminate this? I'm sorry still new to this OS.

What does the rest of the panic say? You might need to use the 'logview' command from the Linode LISH shell to do it (not your system's console):

ssh linodexxxxx@citynameyyy.linode.com logview

Password: (your LISH password, set on the Remote Access page)

I'd bet a substantial portion of the cash in my wallet it will say "Free swap = 0kB" in there. If so, you're running out of memory. I'd also bet on Apache and PHP being involved… if you're using PHP-related stuff and Apache, check your apache2.conf's MaxClients setting in the mpm_prefork section. If it's greater than about 25 or so, you're likely to have a bad time.

(The default is 150, for reasons we'll chalk up to naive optimism guarded by a fierce bureaucracy.)