Sigh! --Another Scheduled Maintenance? Another Unreachable Server

As noted by the output, the file is in /etc/network/if-pre-up.d not /etc/network/if-up.d

OK. So it looks like loads of services on the server are dead. I managed to get networking to come up [I think!] by:

ln -s /etc/alternatives/ip6tables-restore /sbin/iptables-restore

and then

ifup eth0

Now systemctl status networking.service -l --no-pager --full says the network has come up properly:

Sep 16 22:07:12 vostok sudo[1025]:     root : PWD=/ ; USER=root ; COMMAND=/bin/sh -c /sbin/iptables-restore < /etc/iptae
Sep 16 22:07:12 vostok sudo[1025]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 16 22:07:12 vostok sudo[1025]: pam_unix(sudo:session): session closed for user root
Sep 16 22:07:12 vostok sudo[1032]:     root : PWD=/ ; USER=root ; COMMAND=/bin/sh -c /sbin/iptables-restore < /etc/iptae
Sep 16 22:07:12 vostok sudo[1032]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 16 22:07:12 vostok sudo[1032]: pam_unix(sudo:session): session closed for user root
Sep 16 22:07:12 vostok sudo[1064]:     root : PWD=/ ; USER=root ; COMMAND=/bin/sh -c /sbin/iptables-restore < /etc/iptae
Sep 16 22:07:12 vostok sudo[1064]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 16 22:07:12 vostok sudo[1064]: pam_unix(sudo:session): session closed for user root
Sep 16 22:07:12 vostok systemd[1]: Finished Raise network interfaces.`

However, I still can't bring my Caddy webserver up, or start SSH to allow me to login normally. If I do a systemctl start caddy.service or systemctl start ssh.service I just get dumped back to the:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

prompt each time. I don't know what the heck's going on here. This server has been quietly running unattended for months and I've not touched it in a similar amount of time. And then suddenly overnight, it [and my other Linode hosted in the same datacentre] goes down and this one is completely wrecked and acting like its services are broken and looking in the wrong places for their support files.

Anyone from Linode got anything to report on goings on at Frankfurt? It's a bit of a coincidence that both my Linodes in that datacentre suddenly died at the same time, when I've not logged into either, or changed anything on them in weeks, if not months.

So, every service is dead and trying to start any of them dumps me back to that annoying 'emergency mode' prompt:

systemctl list-units --type=service --state=dead


  UNIT                                 LOAD      ACTIVE   SUB  DESCRIPTION
  acpid.service                        loaded    inactive dead ACPI event daemon
  apparmor.service                     loaded    inactive dead Load AppArmor pr>
  apt-daily-upgrade.service            loaded    inactive dead Daily apt upgrad>
  apt-daily.service                    loaded    inactive dead Daily apt downlo>
  atd.service                          loaded    inactive dead Deferred executi>
● auditd.service                       not-found inactive dead auditd.service
  bind9.service                        loaded    inactive dead LSB: Start and s>
  caddy.service                        loaded    inactive dead Caddy
  certbot.service                      loaded    inactive dead Certbot
● clamav-daemon.service                not-found inactive dead clamav-daemon.se>
  clamav-freshclam.service             loaded    inactive dead ClamAV virus dat>
● connman.service                      not-found inactive dead connman.service
  cron.service                         loaded    inactive dead Regular backgrou>
  dbus.service                         loaded    inactive dead D-Bus System Mes>
● display-manager.service              not-found inactive dead display-manager.>
  dovecot.service                      loaded    inactive dead Dovecot IMAP/POP>
  e2scrub_all.service                  loaded    inactive dead Online ext4 Meta>
  e2scrub_reap.service                 loaded    inactive dead Remove Stale Onl>
  exim4.service                        loaded    inactive dead LSB: exim Mail T>
  fail2ban.service                     loaded    inactive dead Fail2Ban Service
  fancontrol.service                   loaded    inactive dead fan speed regula>
● firewalld.service                    not-found inactive dead firewalld.service

UPDATE: 17th Sept and same scenario again. Both Linodes inaccessible again, as of sometime overnight, UK time. I reboot both and one of them comes back online while the other one is still inaccessible with all services dead and LISH console constantly booting me out and back into 'emergency mode' whenever I try to start any services.

Have opened a ticket: #16241056

Let's see if Linode can shed any light on this!

EDIT: This is what happens when I'm logged in via LISH. All vital services are down. But if I try and start any of them, I just get dumped back to the 'emergency mode' prompt and have to login again. Extremely frustrating and tedious!

 ᐅ  service sshd status

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: e>
     Active: inactive (dead)
       Docs: man:sshd(8)
             man:sshd_config(5)

ᐅ  service sshd start
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

@madra --

A couple of things…

• I've been a Linode customer since 2013. Linode never, ever does scheduled maintenance or "tinkering with their infrastructure" without telling you first.

• If it happens two days/nights in a row, it's something else…not "scheduled maintenance".

• From the transcript of your lish session, the culprit is some part of your Linux configuration…not anything to do with your Linode itself. The fact that you got to "emergency mode" in the first place indicates the system booted fine. Unless there's something wrong with networking (which is extremely unlikely), Linode's responsibility ends there.

That's not to say that they won't try to help you out…but, be advised that Linode support folks do not have access to the guts of your distro.

Today's modern Linux distros are little more than a kernel and a collection of systemd(1) services. It looks like you're entering the ring for a cage match with systemd(1). I hope you've stocked up on your blood pressure meds…you're going to need them.

All that being said, there are some indications from the stuff you've posted that your firewall configuration may be the culprit. Can you turn of your firewall completely (for awhile)?

-- sw

From the transcript of your lish session, the culprit is some part of your Linux configuration…not anything to do with your Linode itself.

Why is everyone [including Linode support] ignoring the salient facts here? Let me state them again:

1. This happened simultaneously to 2 different Linodes at the same time

2. Before this down time, I had not logged into either Linode for at least a couple of weeks, so I had not changed anything in their setup

3. The two Linodes are running different OSes [one on Debian, one on Ubuntu]

So how can a configuration error suddenly spontaneously manifest itself on two different Linodes running two different OSes, at the same time, when both Linodes had previously been running for months without issue? And this happen WITHOUT ANY USER INPUT?

Anyway. Some progress has been made. Though how long it lasts is anybody's guess.

Searching for info online I found a few threads wherein people advised that being continually dumped into 'emergency mode' indicated that something had gone wrong with /etc/fstab so that the drives were not mounting properly at boot time.

I checked my /etc/fstab and found that my various drives therein were identified by drive letter, rather than UUID. So, I ran blkid and edited /etc/fstab to use UUIDs instead and then ran mount -a. This time I got a message:

/var/opt/bin/s3fs: /usr/lib/i386-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by /var/opt/bin/s3fs)
/var/opt/bin/s3fs: /usr/lib/i386-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by /var/opt/bin/s3fs)

which relates to two Amazon S3FS drives which I also mount as storage on my Linode. I commented out those lines and ran mount -a again and 'Hey Presto!' my Linode was back online and my websites up again.

So, as I said, we'll have to see whether they stay up this time. But I'm still curious as to what went wrong here?…

1. Did Linode replace some hardware which broke my /etc/fstab since it wasn't using blkids?… or

2. Did `CURL_OPENSSL_3 [whatever that is] get broken somehow. Thus breaking /etc/fstab as regards mounting the S3FS mountpoints?

Again to re-emphasise for about the 3rd time!

Before they both went down, I had not touched either of these Linodes in weeks. So whatever changed to break them [whether hardware or software] was not anything I did.

Why is everyone [including Linode support] ignoring the salient facts here? Let me state them again:

I dunno…s**t happens… I was just trying to help you out. Ranting will only make you feel better. Frankly, it doesn't do anything for me…

Again to re-emphasise for about the 3rd time!
 
Before they both went down, I had not touched either of these Linodes in weeks. So whatever changed to break them

It doesn't have to be related anything that "you did"… Do you have disgruntled ex-employees? Do you have cron(8) jobs to install distro updates automagically? Do you scan your security logs for unauthorized logins and take appropriate countermeasures?

-- sw

Ranting will only make you feel better. Frankly, it doesn't do anything for me…..

Why did you think I was ranting?

It's incredibly frustrating when you ask for support somewhere, taking the time to point out relevant the facts and people respond in a way that suggests they didn't read past the subject line.

And I wasn't singling out you either. I got similar 'advice' from Linode support, suggesting I run fsck on my Linode or update the kernel. When I had also emphasised to them that this problem arose simultaneously on two separate Linodes running two separate Oses, neither of which had been touched my me in weeks, if not months.

Do you have disgruntled ex-employees?

No. I work for myself

Do you have cron(8) jobs to install distro updates automagically?

Now that, at last, is a useful suggestion. However I've not setup any such cronjobs and nor do I have unattended-upgrades setup.

Do you scan your security logs for unauthorized logins and take appropriate countermeasures?

Not yet. But I think unauthorised logins highly unlikely since I login via publickey and have root login and password login disabled.

It's hard to say what caused your problem (if, in fact, you still have a problem)…or who the culprit is.

What I can say is, that in my many years of dealing with Linux, Linux as a platform for doing what I need/want to do has become increasingly unstable. It seems to get worse with every release of every "mainstream" distro. You can probably think of as many reasons as I can about why this might true. In my experience, 9 times out of 10, the culprit has been systemd(1).

To mitigate the fact that systemd(1) is a bloated, not-POSIX-compliant, inescapable POS, I've abandoned use of Linux altogether…everywhere in my personal computing infrastructure (even at home). Consequently, my BP is several points lower.

Maybe that's not the answer for you but you should maybe at least think about that. There are pros and cons of doing this of course so you'll have to weigh those yourself.

Linux is a one-man show (and it's a well-known fact that this guy has a temper)…and I think problems like yours really expose why this engineering model doesn't work.

-- sw

I think it's safe to assume that Linode have been tinkering with their infrastructure again.

The only two things that could/would happen on Linode’s end to cause your VPS(s) to become unavailable would be:

1. The physical server itself has an emergency issue. A ticket is automatically issued in this case.

2. A serious network outage or DDOS attack which, again, would automatically be issued an alert or ticket.

Keep in mind that their service is partially unmanaged, so you are responsible for configuration issues on your end.