Extremely slow connection between private IPs [SOLVED]

Question

Extremely slow connection between private IPs [SOLVED]

general

I have 2 linodes in Dallas. One's a web server, the other a mysql server, and they communicate over the private network. Today I noticed my site wasn't responding. For some reason, it's taking around 30 seconds to set up a mysql connection to the private address of the mysql server:

$ time mysql -h 192.168.130.XX -u xxxx -p
Enter password: 
... (ctrl-D)
mysql> Bye

real    0m30.632s
user    0m0.004s
sys    0m0.001s

$

Substituting the server's public IP address, it connects instantly. The same thing happens with ssh, so I'm pretty sure it's not a mysql issue. In both cases, once the connection is set up, it's fast as usual.

What the hell? Any ideas? I've tried to poke around and debug this, but I'm not sure where to start. My /etc/network/interfaces on both nodes seem fine, and rebooting both of them didn't help.

Edit: plain HTTP connects instantly over the private IP, though.

5 Replies

forum:bdonlan · Answer 1 · Nov. 18, 2009, 3:59 p.m.

forum:bdonlan 15 years, 1 month ago

One possibility is the server (or client) is attempting to do a reverse DNS lookup on the private IP (which fails, because private IPs are private, and have no rDNS). Try disabling it in the mysql server by passing –skip-name-resolve as a command-line option, or add a hostname for your private IPs to /etc/hosts.

forum:funkytastic · Answer 2 · Nov. 18, 2009, 4:11 p.m.

forum:funkytastic 15 years, 1 month ago

bdonlan: You're right, it was reverse DNS timing out. mysql and ssh were attempting to get the reverse of the private IP, and this took 30 seconds to fail. This same config had been working fine for several weeks, so I'm not sure if this reflects a change in behavior with the Linode resolvers. Either way, I fixed it by adding hostnames for both private IPs in /etc/hosts on both nodes.

Thanks to the irc crew for helping me figure it out!

forum:Crisis · Answer 3 · Nov. 18, 2009, 7 p.m.

forum:Crisis 15 years, 1 month ago

This affected me as well today. Any idea what changed this afternoon to cause it?

forum:funkytastic · Answer 4 · Nov. 18, 2009, 8:56 p.m.

forum:funkytastic 15 years, 1 month ago

blackhole-{1,2}.iana.org are repsonsible for the reverse of 192.168.*. Perhaps these servers were down, or there was some connectivity issue between them and linode. Are you in Dallas too?

forum:Xan · Answer 5 · Nov. 19, 2009, 12:02 a.m.

forum:Xan 15 years, 1 month ago

That's funny… I was diagnosing a problem sending email from my phone today, and tracked it down to reverse DNS lookups taking so long that the client gave up and dropped the connection.

Those were attempts to do lookups on two of Sprint's network blocks, 173.96.0.0/11 and 99.200.0.0/13 .

Figuring that out was quite hair-pulling. Reverse DNS on those IPs is now quick again, so things are normal. Could this be related?

Compute

Storage

Networking

Databases

Services

Developer Tools

Industries

Pricing

Community

Engage With Us

Extremely slow connection between private IPs [SOLVED]

5 Replies

Reply

Tips: