Intermittent Can't connect to MySQL server via private LAN

A long shot, but what kernel are you running?

-Chris

Hi, thanks for a reply.

I am running standard Ubuntu 8.04 2.6.24-24 Xen kernel:

$ uname -a

Linux us1 2.6.24-24-xen #1 SMP Fri Sep 18 18:14:17 UTC 2009 x86_64 GNU/Linux

Just did a flood ping test between Linodes and got strange results:

ping -f 192.168.162.72

PING mysql-rw (192.168.162.72) 56(84) bytes of data.

…Warning: time of day goes back (-1098us), taking countermeasures.

Warning: time of day goes back (-1041us), taking countermeasures.

..Warning: time of day goes back (-1043us), taking countermeasures.

..Warning: time of day goes back (-1003us), taking countermeasures.

.

--- mysql-rw ping statistics ---

693417 packets transmitted, 693413 received, 0% packet loss, time 181824ms

rtt min/avg/max/mdev = 0.000/0.216/9.079/0.112 ms, ipg/ewma 0.262/0.217 ms

Never seen "Warning: time of day goes back (-XXXus), taking countermeasures." before… any idea what that might be?

Also note: there is a bit of packet loss - 4 packets out of 693417 seems not much, but this would be roughly the percentage of broken MySQL connections we are seeing.

The time goes back message means the clocks on your two machines are out of sync, do you have ntpd running?

Do you need to run a "non-linode" kernel? I assume you do, since you took the time to set it up…

Regarding the time errors… are you running ntp on both nodes?

Both Linodes run the following cron job:

15 */2 * * * root /usr/sbin/ntpdate 0.pool.ntp.org > /dev/null

so clocks should be in sync.

Don't cron ntpdate, that's the reason you're seeing jumps in time.

ntpd will gracefully adjust your time, ntpdate just smacks it to the current time, which might be in the past, which can really muck things up.

Bottom line… never use ntpdate in a cron job, always run ntpd.

hmmm… the thing is we are running this set up for the last 30 days, and "Can't connect to MySQL server on '192.168.162.72' (4)" errors started showing up only in the last two days. Please see a log file here

@level7:

hmmm… the thing is we are running this set up for the last 30 days, and "Can't connect to MySQL server on '192.168.162.72' (4)" errors started showing up only in the last two days. Please see a log file here

I'm not promising the time issue is related to the MySQL issue, but it's an easy fix, and bad time can cause very unpredictable errors.

Fix the time issue install ntpd (apt-get install npt) then disable the cron.

Wait until the times sync then see if the error goes away.

Also have you changed any firewall settings lately?

We finally managed to solve the problem by switching to "Latest 2.6 Paravirt" kernel. No more "Can't connect to MySQL server on 'serverLANIP' (4)" errors :)

It's even more critical that you set up ntpd in that case. The paravirt kernels don't get time from the host.