pv-grub and kernel 2.6.32-131

No problems here

  Booting 'CentOS (2.6.32-131.21.1.el6.i686)'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-131.21.1.el6.i686 root=/dev/xvda console=tty0 conso
le=hvc0 ro quiet crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KE
YTABLE=us
initrd /boot/initramfs-2.6.32-131.21.1.el6.i686.img

close blk: backend at /local/domain/0/backend/vbd/1939/51712
close blk: backend at /local/domain/0/backend/vbd/1939/51728
close blk: backend at /local/domain/0/backend/vbd/1939/51744
PCI: Fatal: No config space access function found
isapnp: Write Data Register 0xa79 already used
Reserving virtual address space above 0xf5800000
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-131.21.1.el6.i686 (mockbuild@c6b6.bsys.dev.centos.org) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Tue Nov 22 18:21:07 GMT 2011
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  NSC Geode by NSC
  Cyrix CyrixInstead
  Centaur CentaurHauls
  Transmeta GenuineTMx86
  Transmeta TransmetaCPU
  UMC UMC UMC UMC
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020000000 (usable)
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x20000 max_arch_pfn = 0x400000
initial memory mapped : 0 - 033ff000
init_memory_mapping: 0000000000000000-0000000020000000
NX (Execute Disable) protection: active
 0000000000 - 0020000000 page 4k
kernel direct mapping tables up to 20000000 @ 100000-206000
RAMDISK: 00d13000 - 02a05000
0MB HIGHMEM available.
512MB LOWMEM available.
  mapped low ram: 0 - 20000000
  low ram: 0 - 20000000
  node 0 low ram: 00000000 - 20000000
  node 0 bootmap 00002000 - 00006000
(9 early reservations) ==> bootmem [0000000000 - 0020000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0002a88000 - 0002aa0000]   XEN PAGETABLES ==> [0002a88000 - 0002aa0000]
  #2 [0000001000 - 0000002000]    EX TRAMPOLINE ==> [0000001000 - 0000002000]
  #3 [0000006000 - 0000007000]       TRAMPOLINE ==> [0000006000 - 0000007000]
  #4 [0000400000 - 0000bede70]    TEXT DATA BSS ==> [0000400000 - 0000bede70]
  #5 [0000d13000 - 0002a05000]          RAMDISK ==> [0000d13000 - 0002a05000]
  #6 [0002a05000 - 0002a88000]   XEN START INFO ==> [0002a05000 - 0002a88000]
  #7 [0000100000 - 00001e9000]          PGTABLE ==> [0000100000 - 00001e9000]
  #8 [0000002000 - 0000006000]          BOOTMAP ==> [0000002000 - 0000006000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  Normal   0x00001000 -> 0x00020000
  HighMem  0x00020000 -> 0x00020000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00020000
On node 0 totalpages: 130975
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 3967 pages, LIFO batch:0
  Normal zone: 992 pages used for memmap
  Normal zone: 125984 pages, LIFO batch:31
Using APIC driver default
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 4 CPUs, 0 hotplug CPUs
Local APIC disabled by BIOS -- you can enable it with "lapic"
APIC: disable apic facility
nr_irqs_gsi: 16
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000)
Booting paravirtualized kernel on Xen
Xen version: 3.4.1 (preserve-AD)
NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 14 pages/cpu @c2ebd000 s34456 r0 d22888 u65536
pcpu-alloc: s34456 r0 d22888 u65536 alloc=16*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
trying to map vcpu_info 0 at c2ebe020, mfn 62f142, offset 32
cpu 0 using vcpu_info at c2ebe020
trying to map vcpu_info 1 at c2ece020, mfn 62f132, offset 32
cpu 1 using vcpu_info at c2ece020
trying to map vcpu_info 2 at c2ede020, mfn 62f122, offset 32
cpu 2 using vcpu_info at c2ede020
trying to map vcpu_info 3 at c2eee020, mfn 62f112, offset 32
cpu 3 using vcpu_info at c2eee020
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129951
Kernel command line: root=/dev/xvda console=tty0 console=hvc0 ro quiet  SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
Initializing HighMem for node 0 (00000000:00000000)
Memory: 479572k/524288k available (4260k kernel code, 44168k reserved, 2259k data, 524k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xf52d5000 - 0xf57ff000   (5288 kB)
    pkmap   : 0xf4e00000 - 0xf5000000   (2048 kB)
    vmalloc : 0xe0800000 - 0xf4dfe000   ( 325 MB)
    lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
      .init : 0xc0a5e000 - 0xc0ae1000   ( 524 kB)
      .data : 0xc082927b - 0xc0a5dfe8   (2259 kB)
      .text : 0xc0400000 - 0xc082927b   (4260 kB)
Hierarchical RCU implementation.
NR_IRQS:2304 nr_irqs:304
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 2621440 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
Detected 2266.746 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 4533.49 BogoMIPS (lpj=2266746)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 16
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only.
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
Initializing CPU#1
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 2
Initializing CPU#2
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 3
  alloc irq_desc for 16 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 17 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 18 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 19 on node 0
  alloc kstat_irqs on node 0
Initializing CPU#3
CPU: Unsupported number of siblings 16
Brought up 4 CPUs
sizeof(vma)=100 bytes
sizeof(page)=32 bytes
sizeof(inode)=352 bytes
sizeof(dentry)=132 bytes
sizeof(ext3inode)=508 bytes
sizeof(buffer_head)=56 bytes
sizeof(skbuff)=184 bytes
sizeof(task_struct)=1332 bytes
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
  alloc irq_desc for 20 on node 0
  alloc kstat_irqs on node 0
PCI: Fatal: No config space access function found
bio: create slab <bio-0>at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 29640k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
apm: BIOS not found.
audit: initializing netlink socket (disabled)
type=2000 audit(1323540769.547:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 994
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key E197AE6AB18C77CE
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
isapnp: Write Data Register 0xa79 already used
  alloc irq_desc for 21 on node 0
  alloc kstat_irqs on node 0
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
Using IPI No-Shortcut mode
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vbd/51728
XENBUS: Device with no driver: device/vbd/51744
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 524k freed
Write protecting the kernel text: 4264k
Write protecting the kernel read-only data: 1824k
dracut: dracut-004-53.el6_1.1
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
xlblk_init: register_blkdev major: 202 
  alloc irq_desc for 22 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 23 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 24 on node 0
  alloc kstat_irqs on node 0
blkfront: xvda: barriers disabled
 xvda: unknown partition table
blkfront: xvdb: barriers disabled
 xvdb: unknown partition table
blkfront: xvdc: barriers disabled
 xvdc: unknown partition table
kjournald starting.  Commit interval 5 seconds
EXT3-fs (xvda): mounted filesystem with ordered data mode
dracut: Remounting /dev/xvda with -o noatime,errors=remount-ro,ro
EXT4-fs (xvda): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/xvda
dracut: Loading SELinux policy
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1323540770.813:2): selinux=0 auid=4294967295 ses=4294967295
dracut: /sbin/load_policy: Can't load policy: No such file or directory
dracut: Switching root
readahead-collector: starting
udev: starting version 147
Initialising Xen virtual ethernet driver.
  alloc irq_desc for 25 on node 0
  alloc kstat_irqs on node 0
microcode: CPU0 sig=0x106a5, pf=0x1, revision=0x11
platform microcode: firmware: requesting intel-ucode/06-1a-05
microcode: CPU1 sig=0x106a5, pf=0x1, revision=0x11
platform microcode: firmware: requesting intel-ucode/06-1a-05
microcode: CPU2 sig=0x106a5, pf=0x1, revision=0x11
platform microcode: firmware: requesting intel-ucode/06-1a-05
microcode: CPU3 sig=0x106a5, pf=0x1, revision=0x11
platform microcode: firmware: requesting intel-ucode/06-1a-05
Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
EXT4-fs (xvdc): mounted filesystem with ordered data mode
Adding 262136k swap on /dev/xvdb.  Priority:-1 extents:1 across:262136k SS
ip_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (7967 buckets, 31868 max)
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
eth0: no IPv6 routers present
type=1305 audit(1323540802.649:14807): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
type=1305 audit(1323540802.649:14808): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
readahead-collector: sorting
readahead-collector: finished</tigran@aivazian.fsnet.co.uk></bio-0> 

@vchepkov:

  Booting 'CentOS (2.6.32-131.21.1.el6.i686)'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /vmlinuz-2.6.32-131.21.1.el6.i686 ro root=LABEL=root rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=hvc0 crashkernel=auto
initrd /initramfs-2.6.32-131.21.1.el6.i686.img

@Nibbler:

  Booting 'CentOS (2.6.32-131.21.1.el6.i686)'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-131.21.1.el6.i686 root=/dev/xvda console=tty0 console=hvc0 ro quiet crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.32-131.21.1.el6.i686.img

Pretty large differences between these boot lines… both paths (/boot vs /) and parameters (all the NO_ stuff in the crashing one)…

Sounds kinda like one of you installed system afresh and other upgraded, or something?

I have installed it when Centos 6.0 was released, using kickstart and then modified /boot partition to be compatible with Linode's pv-grub, so anaconda created my grub.conf

What I also noticed in my output:

Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (junk in compressed archive); looks like an initrd
Freeing initrd memory: 27124k freed

This is strange.

the file seems perfectly fine to me:

# gunzip -tc < initramfs-2.6.32-131.21.1.el6.i686.img
# echo $?
0

# lsinitrd initramfs-2.6.32-131.21.1.el6.i686.img |head
initramfs-2.6.32-131.21.1.el6.i686.img:
========================================================================
dracut-004-53.el6_1.1
========================================================================
drwxr-xr-x  22 root     root            0 Dec 11 01:21 .
drwxr-xr-x   2 root     root            0 Dec 11 01:21 sbin
-rwxr-xr-x   1 root     root        21300 Jul 20 10:27 sbin/blkid
-rwxr-xr-x   1 root     root        11660 Jul 20 10:27 sbin/blockdev
-rwxr-xr-x   1 root     root          274 Oct 14 04:30 sbin/insmodpost.sh
-rwxr-xr-x   1 root     root         6928 Jul 20 10:27 sbin/switch_root

# ls -l initramfs-2.6.32-*
-rw-r--r--. 1 root root 11524702 Dec 11 01:22 initramfs-2.6.32-131.21.1.el6.i686.img
-rw-r--r--. 1 root root 10367751 Sep  3 16:56 initramfs-2.6.32-71.29.1.el6.i686.img

At this point I think something is wrong with the hosting machine - newark394, I have created another linode with the same disk schema:
> # cat /proc/partitions

major minor #blocks name

202 0 204800 xvda

202 16 30203904 xvdb

202 17 30202880 xvdb1

202 32 1048576 xvdc

and newark405 host it works just fine, I even copied over initramfs from working node to another: no dice, pv_grub now crashes right away with centos6.2 kernel vmlinuz-2.6.32-220.el6.i686

these are the last entries in the lish log before linode get terminated:
> root (hd0)

Filesystem type is ext2fs, using whole disk

kernel /vmlinuz-2.6.32-220.el6.i686 ro root=LABEL=root rdNOLUKS rdNOLVM rd_

NOMD rdNODM LANG=enUS.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=h

vc0 crashkernel=auto

initrd /initramfs-2.6.32-220.el6.i686.img

close blk: backend at /local/domain/0/backend/vbd/3489/51712

close blk: backend at /local/domain/0/backend/vbd/3489/51728

close blk: backend at /local/domain/0/backend/vbd/3489/51744

@vchepkov:

and newark405 host it works just fine, I even copied over initramfs from working node to another: no dice, pv_grub now crashes right away with centos6.2 kernel vmlinuz-2.6.32-220.el6.i686

Open a ticket with support. While linode staff do read the forums, they are not the official support method.

I was told to appeal to the community:

> Greetings Vadym,

Thank you for taking the time to contact us. While we do provide the ability to run your own non-Linode kernel on your Linode using PV_GRUB, troubleshooting issues that arise with this configuration falls outside of our scope of support.

I'd suggest touching base with the community for some help on figuring out why this is not working. The Linode active user community is full of fellow Linode customers many of which are seasoned IT consultants and system administrators. You can find out more about the community here:

• http://www.linode.com/community/

I'm sorry that I cannot be of any direct assistance. If you have any further questions don't hesitate to contact us.

Regards,

Tim

Found this old thread as I'm researching a similar issue. Notice the difference in the two instances described above (one working, one not) is the memory size. This fits with my findings that pg-grub won't boot on an instance with more than 1017MB of RAM.

Nice catch.

Yep, by lowering "Memory limit" in the profile I was able to boot kernel

kernel-2.6.32-220.el6.i686

Definitely a bug in pv_grub.

Had this issue ages ago with backported 2.6.something kernels with Debian, fortunately it doesn't seem to be an issue anymore.

You either need to decrease or increase the RAM, instances with larger amounts of RAM seemed fine, the issue was around the 1024MB range.

@-Alex-:

Had this issue ages ago with backported 2.6.something kernels with Debian, fortunately it doesn't seem to be an issue anymore.

You either need to decrease or increase the RAM, instances with larger amounts of RAM seemed fine, the issue was around the 1024MB range.

So, is this likely to be an issue with the kernels or with pv-grub?

Absolutely no idea, don't really have the skill to analyse and debug this one. Just took me a while to figure out that this only happened on instances with 1024MB of RAM and not higher or lower, was fortunate that I had various servers running with the same distribution and kernel which were working fine.