Clock drift!
Because this is indeed a VPS provider, the host load may drastically slow your clock down. In short, your clock is going to run slow.
If you don't want your clock to run slow, install and run ntpd.
So go install ntpd! Package name might be "ntp" or "ntpd." Be sure to start the service too.
Hey, linode people! Any chance we could get an internal ntp server to sync to? :)
edit: hey linode people! How about opening up the ntp daemon on the hosts to the VMs they support? I like that idea even better.
22 Replies
I had a UML based linode, that is now a paravirt-kernel linode.
This is news to me.
@glg:
do a search in the forums, it's pretty well known. Just grab ntpdate.
I did, and found… very very little. Search for "clock AND drift" and you'll see why I posted this.
@arjones85:
Is there any benefit to running ntpd vs just setting up a cronjob to do "ntpdate pool.ntp.org" every month or something?
The advantage of ntpd is that it continually keeps your clock accurate. Instead of jumping forward (or whatever) a large number of seconds, it continually makes minor adjustments to keep the clock in sync. ntpd also (with most distro-provided config files) syncs with several NTP sources at once, which can help negate the effects of general internet latency.
I was seeing a drift of ~2.4186 seconds per day. ntpdate could have been run twice daily and it would help, but with ntpd, you can forget about it entirely.
Note that you can't run ntpd and ntpdate at the same time, as the NTP protocol uses port 139 exclusively on both the server and the client side.
Most importantly for your system, it will change your clock abruptly, which can make log timestamps ambiguous. If your clock is fast (its time is ahead of the reference time), a sudden jump backwards can freak things out.
It also causes a sudden and heavy load on the time servers at the top of the minute, even more so at the common cron times (*, */2, */3, etc). You're much better off investing the ~1MB of RAM and running ntpd, pointing it at 0.pool.ntp.org, 1.pool.ntp.org, and 2.pool.ntp.org.
This message brought to you by 0.19% of us.pool.ntp.org
(Also, the clock decoupling appears to be a side effect of the mainline Linux Xen implementation. I believe we're seeing the physical host's inherent clock drift. The virtualization is THAT GOOD.)
@hoopycat:
It also causes a sudden and heavy load on the time servers at the top of the minute, even more so at the common cron times (*, */2, */3, etc).
The statistics geek in me would love to see a graph of requests for the top of the minute requests. It would seem that the spike would relate to the amount of drift in clocks for those that request them. Probably more like a prolonged 15 second spike on each side of the top of the minute?
@hoopycat:
(Also, the clock decoupling appears to be a side effect of the mainline Linux Xen implementation. I believe we're seeing the physical host's inherent clock drift. The virtualization is THAT GOOD.)
So my host drifts more than one second per twelve hours? :/
I sure hope not…
pats ntpd
@kbrantley:
So my host drifts more than one second per twelve hours? :/
I sure hope not…
I have a PC that drifts a minute a day. I've heard of worse. There is a threshold where NTP will tell you to get bent and no longer sync because you drift so much, and I've heard of it being hit, too.
@jed:
@kbrantley:So my host drifts more than one second per twelve hours? :/
I sure hope not…
I have a PC that drifts a minute a day. I've heard of worse. There is a threshold where NTP will tell you to get bent and no longer sync because you drift so much, and I've heard of it being hit, too.
Sounds a bit like my router… (which has done that before)
@jeffml:
The statistics geek in me would love to see a graph of requests for the top of the minute requests. It would seem that the spike would relate to the amount of drift in clocks for those that request them. Probably more like a prolonged 15 second spike on each side of the top of the minute?
Rest assured that if I could remember where I've seen the graphs, I would have included them.
Also, never underestimate how accurate your clock will be if you run ntpdate every minute.
(Don't get me started on htpdate.)
@kbrantley:
So my host drifts more than one second per twelve hours? :/
But of course. Your host's hardware clock oscillator deviates from an ideal reference oscillator by about 23 parts per million, or 0.0023%. If we assume a 32768Hz oscillator, it's off by less than 1Hz. That's an easy software fix.
Now, the hard thing to fix is the frequency drift: the rate of change of the frequency is a pain in the butt. ntpd is really good about handling that.
Fun reading: http://www.ntp.org/ntpfaq/NTP-s-sw-clocks-quality.htm
@kbrantley:
Hey, linode people! Any chance we could get an internal ntp server to sync to?
:)
why yes, that is a possibility. This is the short list, there are others.
ISTR that one doesn't let me mess with the clock at all…
@rsk:
Urm, if I can ask for clarification… is NTP necessary only when using paravirt kernels, or with the "oldstyle Xen" 2.6.18.8 too?
ISTR that one doesn't let me mess with the clock at all…
Until you decouple from the host clock by twiddling this knob in /proc:
10:36 jsmith@staff% uname -a
Linux staff.linode.com 2.6.18.8-linode22 #1 SMP Tue Nov 10 16:12:12 UTC 2009 i686 GNU/Linux
10:36 jsmith@staff% cat /proc/sys/xen/independent_wallclock
0
When set to 1, you can set your clock independently of the host.
Native Xen kernels (i.e., 2.6.18 and pals) have this configuration knob, and paravirtualized kernels do not – their clock is automatically decoupled.
@jed:
@rsk:Urm, if I can ask for clarification… is NTP necessary only when using paravirt kernels, or with the "oldstyle Xen" 2.6.18.8 too?
ISTR that one doesn't let me mess with the clock at all…
Until you decouple from the host clock by twiddling this knob in /proc:10:36 jsmith@staff% uname -a Linux staff.linode.com 2.6.18.8-linode22 #1 SMP Tue Nov 10 16:12:12 UTC 2009 i686 GNU/Linux 10:36 jsmith@staff% cat /proc/sys/xen/independent_wallclock 0When set to 1, you can set your clock independently of the host.
Native Xen kernels (i.e., 2.6.18 and pals) have this configuration knob, and paravirtualized kernels do not – their clock is automatically decoupled.
Dumb question- do you simply edit the file (cat /proc/sys/xen/independent_wallclock ) and reboot?
@bryantrv:
Dumb question- do you simply edit the file (cat /proc/sys/xen/independent_wallclock ) and reboot?
No, since /proc will be recreated at boot. Set
xen.independent_wallclock = 1
in /etc/sysctl.conf then run "sysctl -p". You do not have to reboot; it will be set on subsequent boots.
@kbrantley:
Hey, linode people! Any chance we could get an internal ntp server to sync to?
:)
Aside from jtsage's link, there are other public NTP servers hosted on Linodes, too; some are in the. stratum 2 list
@kbrantley:
edit: hey linode people! How about opening up the ntp daemon on the hosts to the VMs they support? I like that idea even better.
It already is. Try it. There are no service guarantees, though.
@jeffml:
The statistics geek in me would love to see a graph of requests for the top of the minute requests. It would seem that the spike would relate to the amount of drift in clocks for those that request them. Probably more like a prolonged 15 second spike on each side of the top of the minute?
I don't have any graphs, but in my experience, the spike peaks about 4 seconds after the hour. Traffic may have started increasing a few seconds before the hour, but not very significantly. It stays unusually high for at least a few minutes, and, of course, there are smaller spikes every minute. (The spike 1 minute after the hour is pretty significant, too.)
You could also add time.xenscale.com as well. Its a Linode in Dallas and a pool contributor.
@mnordhoff:
@kbrantley:edit: hey linode people! How about opening up the ntp daemon on the hosts to the VMs they support? I like that idea even better.
It already is. Try it. There are no service guarantees, though.
Check that out.
is perfectly happy