Networking issues after recent maintenance. It does not seem the Network Helper is at fault.

Hi,
It sounds like your network interface eth0, given to you by Linode, is acting like a network interface that is either not connected to anything, or simply isn't obtaining any IP addresses through any configuration. As another command to temporarily help isolate any issues with a dhcp server on Linode's end, or perhaps the host network configuration, you could run this command as root.
dhcpcd eth0
Does this give you an ip? You could then run this to show the configuration.
ip addr show dev eth0

Blake

I personally have known Docker to screw about with networking config in order to set up its own bridge etc., so I wouldn’t put it past it to be interfering.

Do you have the network helper enabled now?

It does look suspicious that the “orig” and “last” files have DHCP enabled, but the “live” one does not. The “live” one is re-generated on every boot.

Try adding the additional “Network” block from the Linode-orig file into the 05-eth0.network file and reboot. If that gets removed after a reboot, either something is seriously off with the network helper and it’s generating an incorrect config file, or something is screwing with it post-boot after the helper has created it.

Alternatively, have you tried configuring your networking as static according to this guide? Disable the network helper in your profile and pay heed to the notice about removing the previously-generated files so as not to interfere with the static IP config.

You can get all the details (IP, gateway, DNS etc) from your Linode’s Networking tab in Manager.

Reboot and your network config is now fully in your hands!

Hi Andy and Blake,

First of all thanks a lot for taking the time to help me out on this!

Here's the output of the commands Blake mentioned:

$ dhcpcd eth0
-bash: dhcpcd: command not found

$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:3c:92:f9:02:9e brd ff:ff:ff:ff:ff:ff

For some reason I am getting a lot of commands not found… And as the node does not have a working connectivity I cannot download any missing packages.

So to come to Andy's suggestions: yes, the support team also suggested I disable the Network Helper and manually setup a static IP. I followed the guide you linked up to the last command (for Ubuntu >18.04).

Unfortunately, the last command throws a similar 'not found' error:

sudo: netplan: command not found

That was prior to restoring the backup linode with the Network Helper disabled from the get-go.

Seems like I am stuck in every way, should I set up a completely fresh node? I would just like to know enough to prevent something similar from happening during a future maintenance.

Hmm something looks very odd - Ubuntu 18.04 on uses Netplan for its networking config by default, so that command being missing suggests it has been uninstalled at some point.

Ok let’s try something else. Netplan generates systemd network config files. Here’s one of mine. Change the values accordingly to what’s in your Linode’s network tab (put this in /etc/systemd/network/01-eth0.network, remove the Netplan one and reboot.)

[Match]                                                                                      
Name=eth0                                                                                    

[Network]                                                                                    
LinkLocalAddressing=ipv6                                                                     
Address=your.public.ip/24                                                                     
Address=your.private.ip/17                                                                                                                         
Address=your:ipv6:ip/64                                            
Gateway=your.gateway.ip                                                                         
Gateway=fe80::1                                                                              
DNS=178.79.182.5                                                                             
DNS=176.58.107.5                                                                             
DNS=176.58.116.5                                                                             
DNS=176.58.121.5                                                                             
DNS=151.236.220.5                                                                            
DNS=212.71.252.5                                                                             
DNS=212.71.253.5                                                                             
DNS=2a01:7e00::5                                                                             
DNS=2a01:7e00::6                                                                             
DNS=2a01:7e00::7                                                                             
DNS=2a01:7e00::8                                                                             
DNS=2a01:7e00::9                                                                             
DNS=2a01:7e00::b                                                                             
DNS=2a01:7e00::c                                                                             
Domains=members.linode.com

That's really odd indeed, can docker or remove packages like that? I would assume not.

I removed the netplan generated file which was named 05-eth0.network, replaced it with the template you provided and rebooted. Still no luck…

Could it be that config file is not read because something else is interfering? There is another (empty) file in that directory called 99-default.link, is that supposed to be there?

@SteliosR

I’ve just found this community post which also may have some good advice in it.

In particular, the commands to bring up the interface statically, which may get you back online to allow you to re-install netplan, so you can follow the static IP guide.

sudo ip addr flush dev eth0
ip link set eth0 up
ip addr add 198.51.100.5/24 broadcast 198.51.100.255 dev eth0
ip route add default via 198.51.100.1

Hi Andy,

I read through the guide and attempted those commands still without luck. How can a maintenance screw up the network configs so bad?

I am completely out of my depth here as it's my first time setting up any type of server and networking configuration seem really confusing.

So far, it seems the Network Helper has been ruled out as a possible cause of the problem if I understand correctly. So I restored the backup again, this time with the Network Helper still on and without switching IPs. It didn't work either.

Out of curiosity I also pinged 192.168.147.1 and it returns Destination Host Unreachable. How can I possibly LiSh into my node if it cannot even communicate in its internal network?

Also I've got some important documents on there. Even if I end up doing a fresh install on a new linode, how could I copy those over to another device?

Thanks for the continued help.

How can a maintenance screw up the network configs so bad?

On the basis that manually bringing up the network doesn’t seem to work either, I don’t think this is anything to do with the maintenance, other than the maintenance has triggered a full reboot of your Linode.

The network helper pretty much sorts out any networking issues by making sure the network config is correct after each reboot, but it’s effectively only doing what we’re trying to do manually.

If that doesn’t work, there is something amiss on your Linode.

Can you post the full terminal output after running the commands I suggested above, substituting your IP addresses.

After that, post the output of “ip a” and “ip r”.

How can I possibly LiSh into my node if it cannot even communicate in its internal network?

LISH does not communicate over a network. It’s the actual console of your server and is designed to give you full access to your Linode even if the whole networking stack is down.

The private IP (192.168……) is for your Linode to communicate with another Linode in the same datacentre.

To SSH into your Linode, you need to use your Linode’s public IP.

I see, thanks for clearing that up. The missing packages is the most mysterious part. As a side note, the only other program I can think of that might have interfered with the network configurations is Caddy2, the webserver, is that something worth looking into?

Also, here's the full output for the commands again as you requested:

ubuntu@localhost:~$ sudo ip link set eth0 up
ubuntu@localhost:~$ sudo ip addr add 198.51.100.5/24 broadcast 198.51.100.255 d0
ubuntu@localhost:~$ sudo ip route add default via 198.51.100.1
ubuntu@localhost:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defau0
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP gro0
    link/ether f2:3c:92:f9:02:9e brd ff:ff:ff:ff:ff:ff
    inet 198.51.100.5/24 brd 198.51.100.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2a01:7e01::f03c:92ff:fef9:29e/64 scope global dynamic mngtmpaddr 
       valid_lft 2591984sec preferred_lft 604784sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DO 
    link/ether 02:42:ad:ae:43:28 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-908470d1cb57: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue  
    link/ether 02:42:d0:c4:82:9e brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-908470d1cb57
       valid_lft forever preferred_lft forever
ubuntu@localhost:~$ ip r
default via 198.51.100.1 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.22.0.0/16 dev br-908470d1cb57 proto kernel scope link src 172.22.0.1 linkdo 
198.51.100.0/24 dev eth0 proto kernel scope link src 198.51.100.5

It looks like you’ve used the exact IPs from the commands I’d posted from the linked article.

You do need to replace the IP address (198.51.100.5) and gateway (198.51.100.1) with those assigned to your Linode (you can find these from your Linode’s networking tab in Manager.)

Reboot and try these commands again but with your Linode’s assigned details, and then check the connectivity again.

Hi Andy,

My bad, I was not entirely sure whether to change them or not.

Here's the output again:

ubuntu@localhost:~$ sudo ip addr flush dev eth0
[sudo] password for ubuntu: 
ubuntu@localhost:~$ sudo ip link set eth0 up
ubuntu@localhost:~$ sudo ip addr add 172.104.247.21/24 broadcast 172.104.247.255 dev eth0
ubuntu@localhost:~$ sudo ip route add default via 172.104.247.1

And after reboot:

ubuntu@localhost:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defau0
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qle0
    link/ether f2:3c:92:f9:02:9e brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DO 
    link/ether 02:42:ab:6c:76:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-908470d1cb57: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue  
    link/ether 02:42:a9:ad:6a:54 brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-908470d1cb57
       valid_lft forever preferred_lft forever
ubuntu@localhost:~$ ip r
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.22.0.0/16 dev br-908470d1cb57 proto kernel scope link src 172.22.0.1 linkdo

I meant reboot to clear out any previous config, then run those commands to manually allocate the IP address and bring up the interface, so you can install Netplan, or use SFTP from a new Linode to transfer files across.

Rebooting after running those commands will have just reset it back to a non working state.

I notice your username is “ubuntu”. Some images of Ubuntu (like those made for OpenStack) create an “ubuntu” user to dissuade people from logging in as root. Linode’s images do not; they give you root.

Was this Linode built from a Linode image or a custom image? If it was a custom image it might explain why the network helper is not working for you.

Alright, I did it correctly this time and managed to get connectivity!

I installed netplan, disabled the Network Helper and went over the guide again.

I removed other netplan config files so they don't interfere either. At that point it was not working after calling netplan apply.

Then I remembered you said netplan generates config systemd config files, so I also removed the systemd config file I had added manually and ran sudo netplan apply again.

Connectivity worked after that! Although weirdly, there was no new systemd config file. So I rebooted to make sure the changes persisted, and it all seems fine after reboot as well. So glad it's back up and running, thanks a lot for all your help!

Was this Linode built from a Linode image or a custom image? If it was a custom image it might explain why the network helper is not working for you.

It was built from a Linode image, I set up the ubuntu user myself. Does the Network Helper get confused by non-root users?

Fantastic, so glad you're up and running again.

It was built from a Linode image, I set up the ubuntu user myself. Does the Network Helper get confused by non-root users?

No, not at all, I just know from experience some cloud images like those used on OpenStack clouds set up a user called "ubuntu" by default, but Linode's images don't.

Therefore the presence of this "ubuntu" user might have indicated it was a custom image - which would possibly have confused the Network Helper. But as you set up the ubuntu user yourself, it's fine.

I see, it would have been nice to know for sure what was the root of the problem to avoid doing any mistake twice. At least now it seems a reboot doesn't break things so a future maintenance should be fine.