My Linode is unreachable after maintenance

Linode Staff

This is a basic troubleshooting guide for helping our customers that have issues with accessing their Linode after Maintenance is applied.

Is the Linode powered on?

To see if your Linode is running, you can take a look at your Linode Manager’s dashboard and look for “Running” under the “Status” section. If the status is “Powered Off”, just click on the your Linode’s name in blue, then click the grey Boot button.

Lish

In some cases, the Linode will boot, but there is no SSH access. Our Lish console should be available even when SSH is not. Lish can be used to run any troubleshooting commands that might offer some insight into what the issue is.

You can access your Lish console from the “Remote Access” tab on your Linode’s dashboard. To access Lish through your browser, click “Launch Lish Console”. You will then be able to input your login credentials. As a note, Lish is out of band, so you can use your root login credentials if needed.

https://www.linode.com/docs/networking/using-the-linode-shell-lish/

Network Helper

Our Network helper ensures that your network interfaces are brought up correctly after a reboot by automatically statically assigning your IPv4 address. In your Linode’s configuration profile, turn on Auto-Configure Networking, then click Save. Once that is complete, you can reboot the Linode.

https://linode.com/docs/platform/network-helper/

Checking Interfaces/Networking

There are a number of commands that we can run to determine if a Linode’s networking interfaces are brought up properly after a reboot.

Debian/Ubuntu

cat /etc/network/interfaces
systemctl status networking.service
ip a
ip r

CentOS

cat /etc/sysconfig/network-scripts/ifcfg-eth0
systemctl status -l  network.service
ip a 
ip r

[FAILED] Failed to start Raise network interfaces.

If you see the following error message there are specific commands that you can run to see what might have caused the error.

[FAILED] Failed to start Raise network interfaces. See 'systemctl status networking.service' for details.

Commands:

sudo systemctl status networking.service -l
sudo journalctl -u networking --no-pager | tail -20

If you run those commands and see an output similar to the following, there could be a specific firewall issue that is occurring.

Apr 06 01:03:17 xlauncher ifup[6359]: run-parts: failed to exec /etc/network/if- Apr 06 01:03:17 xlauncher ifup[6359]: run-parts: /etc/network/if-up.d/iptables e

You can run the following commands to resolve this issue.

sudo mv /etc/network/if-up.d/iptables ~
ifdown -a && ifup -a

Please note that your firewall will be down at this point, so you will need to re-enable it manually.

Do you have SendMail installed?

If you have SendMail installed and see the following error message, you might be experiencing an issue with a broken SendMail update

[FAILED] Failed to start Raise network interfaces.^M See 'systemctl status networking.service' for details.^M

Running the following commands from your Linode, while connected via Lish, will give us some more information about the problem:

systemctl status networking.service -l
sudo ifdown -a && sudo ifup -a

If you run those commands and see output similar to the following, it is likely that the broken update is at fault.

/etc/network/if-up.d/sendmail: 44: .: Can't open /usr/share/sendmail/dynamic run-parts: /etc/network/if-up.d/sendmail exited with return code 2

The sendmail issue can usually be resolved by running the following two commands:

sudo mv /etc/network/if-up.d/sendmail ~
ifdown -a && ifup -a

You can read more about the SendMail bug at the following link.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873978

Did all of your services start after the reboot?

Another troubleshooting step that is useful involves determining what services are currently running on your Linode. The following commands will show you what services are running.

sudo netstat -plntu
systemctl list-units --type=service
service --status-all

If you find that your web server is not running, you will want to restart the service. In most cases, your Linode will be running either Apache or Nginx.

Apache

Debian and Ubuntu:

sudo service apache2 restart

Fedora and CentOS:

sudo service httpd restart

Nginx:

sudo service nginx restart

You might also find that your MySQL service is not running. The following command will help you with restarting that.

sudo service mysqld restart

SSH service did not start

If your SSH service did not start, you can run this command to bring it back up.

sudo service ssh restart

or

sudo systemctl restart sshd.service

iptables

If everything seems to be running as expected but you are still unable to connect, you might be running into issues with your firewall. Running the following command will show your your iptables ruleset

sudo iptables-save

If you are unable to determine if a specific rule is causing issues, you can flush your rules and start over. The first command creates a temporary backup of your iptables rules that you can restore if flushing does not resolve the issue.

sudo iptables-save > /tmp/iptables.txt 
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -F
sudo iptables -X

8 Replies

Was your interface renamed?

If your interfaces have been renamed to ensp0*or something other than eth0, this is likely due to the latest version of systemd (226-1+), which uses "Predictable Network Interface Names".

You can disable the use of Predictable Network Interface Names with the following command:

ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules

After running the command, please reboot your Linode.

For additional details about Predictable Network Interface Names, you can refer to this link.

https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

If the above ln command does not work, according to the freedesktop.org wiki link above, the way to disable the predictable network interface names is to use the following (you need to reboot):

ln -s /dev/null /etc/systemd/network/99-default.link

If you're not able to ping your Linode's IP, see if you notice the following error message in your boot log or when running the commands "service network restart" or "ifup eth0":

Bringing up interface eth0:  Device eth0 does not seem to be present, delaying initialization.

This can generally be fixed by running the following command and rebooting your Linode:

rm -f /etc/udev/rules.d/70-persistent-net.rules

On older Linodes, Network Helper (auto-configure networking) can have a hard time identifying the distribution. You'll know this happened by seeing the following in your Linode's job history:

Network Helper did not run: could not reliably determine distribution or distribution version

If this happens, you should still be able to get your networking back up with static networking.

In cases of some older versions of Debian, you may find that Network Helper isn't helping. You should review the log from the latest boot job, by running:

sudo less /var/log/boot.log

If you see one of the following outputs, you're in luck, because we may have a fix for you, as it's most likely related to this Debian bug:

3: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether f2:3c:91:df:3a:34 brd ff:ff:ff:ff:ff:ff

From this message, you can see that your eth0 is DOWN. Chances are, the following command might get you back up and running:

ip addr flush dev eth0
ifup eth0

If not, you'll need to configure your Linode's network statically by editing /etc/network/interfaces

As a third and final option, you can run the following commands, as this works on most distros:

sudo ip addr flush dev eth0
ip link set eth0 up
ip addr add 198.51.100.5/24 broadcast 198.51.100.255 dev eth0
ip route add default via 198.51.100.1

I also had a similar issue where I was not able to connect with my Linode server using SSH.

WHat fixed the issue was:

First uninstall the SSH (https://askubuntu.com/questions/656713/uninstalling-ssh)
Then Install the SSH
(https://www.cyberciti.biz/faq/ubuntu-linux-install-openssh-server/)

And yo, I was again able to connect with SSH.

Linode Staff

In addition to mwildman's response, we have a separate Community Questions post that explains network interface renaming: https://www.linode.com/community/questions/18973/lost-networking-after-reboot-network-interface-renamed

There are a few more solutions within that thread that you may find helpful.

In addition to the SendMail issue discussed in this article, there is another SendMail issue. To test for this issue, run this command in your Lish console:

`sudo ifup eth0`

if the result is like this:

/etc/network/if-up.d/sendmail: 44: .: Can't open /usr/share/sendmail/dynamic
run-parts: /etc/network/if-up.d/sendmail exited with return code 2
Failed to bring up eth0.

then your problem is probably a vestige of an old sendmail installation which was not fully removed. Assuming you no longer use sendmail, run the two commands at the bottom of this blog post. My Ubuntu 16.04 site was back online instantly, ending this nightmare.

Here is another thing to be aware of as a possible side effect from Emergency Maintenance on services. In a few instances, storage volumes have been rendered unmountable when the Linode was suddenly rebooted, like in the case of an Emergency Maintenance reboot.

Here is a quick solution that has worked for customers who have experienced this problem in the past:

1 - Detach the Volume
2 - Clone the Volume
3 - Reattach the original Volume (and test that everything is okay)
4 - Delete Volume clone

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct