Warning: Time moved backwards by 314 seconds.

Xen provides the initial clock point to the Linode. That clock is initially based on the host's time, but it can skew over time and there isn't really a good way to stop that. All our hosts keep their clocks in sync with NTP; that doesn't stop Xen from skewing the initial boot time under some circumstances. There really isn't another fix for this besides running NTP inside your Linode. If you have services that don't properly start before NTP has updated your Linode's internal clock, you may want to tune your init system to start them only after NTP has done its thing.

• Les

@akerl:

Xen provides the initial clock point to the Linode. That clock is initially based on the host's time, but it can skew over time and there isn't really a good way to stop that.

Hi Les, thanks for the reply.

As I noted in my post the clock is 5 minutes fast immediately upon reboot of my Atlanta Linode. To prove it's not my setup that is the problem, I booted from Finnix and it also reports the time as off by five minutes.

@akerl:

All our hosts keep their clocks in sync with NTP; that doesn't stop Xen from skewing the initial boot time under some circumstances. There really isn't another fix for this besides running NTP inside your Linode.

When you say under some circumstances, do you know of certain scenarios that yield worse time skews than others? I currently run a 32-bit OS and use a distro kernel via pv_grub. As part of the guide I originally followed I disabled the Xenify Distro option. Does that have any bearing?

@akerl:

If you have services that don't properly start before NTP has updated your Linode's internal clock, you may want to tune your init system to start them only after NTP has done its thing.

• Les

No offense to you, but that seems more like a workaround than an actual fix to the problem. I'll look modifying the init job order like you suggested, but I can't help but feel something is being overlooked.

No, the kernel choice, architecture, or other options internal to your Linode don't impact this. The reality is that the clock used by the hypervisor to give guests their initial time can and will skew over time. We run NTP, which corrects the dom0's clock, not the hypervisor.

• Les

You really ought to run ntpd inside your linode, since your linode's clock will drift otherwise. This will ensure that your clock is always accurate regardless of what's going on on the host system. If you get a big skew right on boot, you can set ntpdate to run on boot, which will give you the gross correction, and then rely on ntpd to cancel out any drift (ntpd basically adjusts how fast time passes to counter drift, without making any sudden adjustments).

@Guspaz:

You really ought to run ntpd inside your linode

I know it's long, but the OP mentions that I have been running ntpd in the Linode for years already. The problem I have with the workarounds mentioned thus far is finding a way to get ntpdate to run before all other services that depend on reliable time.

Dovecot has an Upstart job, it looks like Postfix and ntpd have init.d scripts and the ntpdate command is run as soon as eth0 comes online (/etc/network/if-up.d/ntpdate).

I have not had time to work on this again today, but i was going to try calling ntpdate from rc.local. I do not recall where that falls in the boot order, but I figured it was worth a try before I had to try and write an Upstart job (which I have no experience with).

For the record, this is Ubuntu 12.04 that I am working with.

I'm not sure I understand the problem then. The host system runs ntpd too. The only way you'd see a time jump is if your linode has drifted, which it normally would. If you also run ntpd on your linode, it won't drift, so there won't be any time jump when you reboot.

@Guspaz:

I'm not sure I understand the problem then. The host system runs ntpd too.

That's what I was told via the support ticket. They also mentioned that dom0 runs ntpd, but the hypervisor is handing over the wrong time to the guest. I don't have Xen training, so I don't know the different layers that make up a Xen infrastructure. Googling around I saw mention of some older versions of Xen having a bug which causes the problem described here, but I've no way of knowing how Linode staff have their host systems configured.

As I also mentioned in this thread I booted the VM using Finnix which had the same initial time that my VM's current OS has: 5 minutes too fast. Since I'm unable to set the equivalent of BIOS/CMOS time like I would in a VMware or VirtualBox VM, this tells me the problem lies with the host configuration. In this situation I would suspect that either a software or a misconfiguration issue is to blame. Linode Support says otherwise.

@Guspaz:

The only way you'd see a time jump is if your linode has drifted, which it normally would. If you also run ntpd on your linode, it won't drift, so there won't be any time jump when you reboot.

My earlier comments address this, but for whatever reason (again, my ignorance of Xen doesn't help things) the current time isn't saved properly on shutdown/reboot like it would on another setup (VMware/VirtualBox/real hardware), so the time given by the host (or hypervisor as I've been told) is what the VM has to start with when it boots.

I noticed that there are Upstart jobs dealing with loading and saving the system clock on startup/reboot, but they reference a Real Time Clock device that it appears Xen doesn't support.

All in all, the time given to my Linode on boot is incorrect and to work around it I have to have ntpdate run before any other service that requires accurate time (all of them?). Until I can figure that out I'll continually have problems every time the box is rebooted.

I had the same problem on one node. You could try asking Linode to move you to another host, I've only see this happen once so odds are you'll move to a node that has the correct time or a time skewed in the past which should cause less problems.

@obs:

I had the same problem on one node. You could try asking Linode to move you to another host

That's a good suggestion. Is that what you did to resolve your issue? I'm a little surprised that Linode Support hasn't made that offer to me as a workaround for the problems with their setup.

In my testing so far it is looking like I'd have to use a combination of a new init.d script and modifications to the Dovecot Upstart job to ensure that correct time is set before Postfix, Dovecot and the like come online. It seems that Dovecot is the only daemon to raise such a fuss, but reading over their notes on the issue I imagine that a backwards time jump can't be good for other daemons either.

Moving to a new host is not a solution to this issue. Any other host is just as likely to experience this issue, and statistically all hosts will eventually experience this issue.

• Les

I didn't move host I just ignored it since it only caused issues with fsck on boot up which I could work around by running fsck via finnix. However even though Les states that moving host isn't the solution I've only encountered this once and I've had a lot of nodes over the years.

As a workaround to the hypervisor issue giving the wrong time I've put together an Upstart job, but it still needs work. It successfully sets the time prior to starting up key daemons, but on reboot it appears to hang the system. I've tracked that down to me using the 'rc' Upstart job as a start event.

Anyone know how to safely key off of that job?

While I already have what I need to workaround just Dovecot having the wrong initial time I'd like to correct the issue for all of the daemons that I can.

What I had:

start on (
    runlevel 2 
    and static-network-up 
    and starting cron 
    and starting mysql 
    and starting dovecot 
    and starting ssh
    and starting rc
)

What I have now:

start on (
    runlevel 2 
    and static-network-up 
    and starting cron 
    and starting dovecot 
    and starting ssh
)

I removed rc because of the hanging issue and I removed mysql because Postfix gets upset when it starts via the SysV script and Upstart hasn't yet started the MySQL instance it needs to resolve addresses. Once I find a safe way to key off of rc so I can delay the SysV scripts I'll add it back to the list to delay that daemon's launch as well.

I tried creating a wrapper around the ntpdate command (only run on runlevel 2), but didn't have any luck. I thought about creating a SysV script that only paused execution of the other scripts there, but didn't think it would have any effect.

I keep telling myself that this should be really simple.

Thanks in advance!

Just curious, how often are you rebooting to make this such a big problem - and why?

@vonskippy:

Just curious, how often are you rebooting to make this such a big problem - and why?

Mainly just when applying kernel, OpenSSL or similar patches. However I've had unexpected results because of the time jumps and I'm tired of it, so I'd like to go ahead and knock out a solid workaround and be done with it.

If you have any tips/suggestions I'm all ears. Thanks.

Alright, I'm finally at a point where I feel I have this "good enough" so I can move on to other things. I'm sharing my findings here in case they can help someone else that may be affected by this in the future. Particularly as Les mentioned (paraphrasing) all of their hosts will likely have this problem eventually.

First, I nuked the contents of the /etc/network/if-up.d/ntpdate file as we don't need ntpdate to be called twice, particularly within a short period. When left in place I noticed that the clock was adjusted by X amount twice, which sent the clock into the past 2x what was needed. Once I removed the contents of that file I replaced with:

#!/bin/sh

# Comments here describing why the file was gutted

exit 0

Second, I dropped this Upstart job file into the /etc/init/ directory as /etc/init/ntpdate.conf. The minified version:

description "Sync clock prior to launching time-sensitive services"

task

start on (
    startup
    and static-network-up 
)

script
    /usr/sbin/ntpdate -b -s ntp.ubuntu.com
end script

what I'm using:

# $Id$
# $HeadURL$

description "Sync clock prior to launching time-sensitive services"

################################################################################
# References:
#
#   http://upstart.ubuntu.com/cookbook/#run-a-job-before-another-job
#   http://upstart.ubuntu.com/cookbook/#start-depends-on-another-service
#   http://upstart.ubuntu.com/cookbook/#start-must-precede-another-service
#   http://upstart.ubuntu.com/cookbook/#really-understanding-start-on-and-stop-on
#   http://upstart.ubuntu.com/cookbook/#method-2
#   http://upstart.ubuntu.com/cookbook/#task
#   http://askubuntu.com/questions/21378/how-can-i-make-sure-one-upstart-job-starts-before-other-upstart-jobs/22110#22110
#   http://superuser.com/questions/502536/how-to-depend-on-an-upstart-job-from-an-init-script-on-ubuntu-12-04
#
# Set a VMware Player VM to NOT sync time between host/guest so I could set
# guest's clock to a false value so time syncing would be forced
#
#   http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189
#
#   http://askubuntu.com/questions/62812/why-isnt-my-upstart-service-starting-on-system-boot
#   http://askubuntu.com/a/71426
################################################################################

# Execute this Upstart job at startup and once all network interfaces listed in 
# /etc/network/interfaces have been brought online.

task

start on (
    startup
    and static-network-up 
)

# Debug output to prove that this Upstart job is executing properly
pre-start script
  logger -p daemon.info -t "ntpdate.conf" "pre-start for ntpdate.conf"
end script

post-start script
  logger -p daemon.info -t "ntpdate.conf" "post-start for ntpdate.conf"
end script

script
    logger -p daemon.info -t "ntpdate.conf" "About to execute ntpdate"

    # Use hard-coded NTP server instead of parsing /etc/ntp.conf
    /usr/sbin/ntpdate -b -s ntp.ubuntu.com

    # Parse NTP config file to sync against multiple servers. Borrowed from 
    # /etc/network/if-up.d/ntpdate
    #/usr/sbin/ntpdate-debian -s -b 2>/dev/null || :

    logger -p daemon.info -t "ntpdate.conf" "Just finished executing ntpdate"
end script

On my system this still leaves these four daemons starting with the wrong time, but unless they begin to show abnormal behavior I'm just going to leave things well enough alone (too much else to worry about):

• postgrey

• opendkim

• mysql

• slapd

I hope this helps someone.