Weird problem with Linode private network

Hello fellow Linoders!

I'm experiencing very strange behaviour and was wondering if anybody here can point me what's wrong with my thinking.

I have a couple of linodes. I want to deploy database cluster on those Linodes (postgres-xl in this instance). On each Linode I have public IP as well as private IP. I want to have my cluster talking via private IP.

I have built clean linux boxes (arch linux) on those Linodes. For testing purposes those systems have nothing on them, not even iptables.

I test if cluster will communicate on public interface first. So far so good - no errors, I see happy logs.

I move my cluster to use private network, and at this point I start seeing weird logs suggesting something is going on which I don't quite understand.

5:2544785216:2017-09-12 15:46:20.389 UTC -LOG:  Started to run as GTM-Active.
LOCATION:  main, main.c:740
1:2536441600:2017-09-12 15:46:25.219 UTC -LOG:  could not receive data from client: Bad file descriptor
LOCATION:  pq_recvbuf, pqcomm.c:524
2:2536441600:2017-09-12 15:46:25.219 UTC -FATAL:  Expecting a startup message, but received �
LOCATION:  GTM_ThreadMain, main.c:1098
3:2536441600:2017-09-12 15:46:25.219 UTC -LOG:  could not send data to client: Bad file descriptor
LOCATION:  internal_flush, pqcomm.c:846

I have done nothing apart from changing IP addresses (to use private addresses rather than public). I am using IP addresses assigned to my Linode via web management service.

I have spent a week trying to find out what's going on. Google is not very helpful, also I am not sure where to look for more information. I have compared tcpdump output when postgres-xl was configured on public interfaces with output when postgres-xl was configured on private interfaces. When configured on public interfaces I can see packets with RST,ACK flag and those packets are missing when configured on private interfaces. This is what made me thinking maybe there is some filtering done on infrastructure level within private network.

Is it possible that there is something going on private network that is preventing my cluster from being happy cluster?

2 Replies

are you able to ping between the linodes on the private network addresses?

did you setup the private IP addresses? If you used the network manager thing (it's newer, I'm not super familiar with it, never used it), I think that sets up your private IP, but if you're just using DHCP to get your IP, that'll only get your public IP, you need to separately bring up the private IP.

Hi, yes - all linodes can ping each other via private interface. In fact cluster >>starts<< to initialise and I can see communication is established (sniffed with tcpdump) however at some point I get "Bad file descriptor" error (which google says is related to closed socket). So something on private network is closing the connection in the middle and servers cannot complete clustering.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct