Docker won't start using the latest Linode kernel
I recently rebooted my Linode and Docker will no longer start. I have my Linode configured to boot with the latest 64bit kernel. Prior to rebooting, Docker was starting without error. How do I fix this?
12 Replies
Updated 8.13.18:
Our team is aware of the issue with Docker and our recent kernels:
4.17.14-x86-linode132
and 4.17.14-x86_64-linode113
A specific module for OverlayFS was not present in these kernels which caused incompatibilities with Docker. This module is now included in our latest revisions, 4.17.14-x86-linode133
and 4.17.14-x86_64-linode114
.
There are 2 easy solutions to get Docker back up and running:
- Reboot your Linode to boot with the latest Linode kernel.
Currently the latest 32bit and 64bit Linode kernels include support for OverlayFS.
Make sure your Linode is configured to boot using our Latest kernel within your configuration profile. This selection will allow your Linode to boot using the latest stable Linode kernel at each reboot.
- Use your distribution's upstream kernel.
Since all Linodes run on KVM, they are able to boot using either a Linode supplied kernel or your distribution's upstream kernel. You may have noticed that the newest distributions are already configured to use GRUB2 and your distribution's upstream kernel. This is the now the default method for all new Images and the preferred selection for kernels. To boot using a distribution supplied kernel, follow the steps in our guide here:
Running a Distribution-Supplied Kernel
Why would you deliberately release (and make default) a kernel which breaks a widely used piece of software, without announcing it?
It also breaks networking on a Debian 7 server I manage.
I saw this today with 4.18.16-x86_64-linode118
Switched to using the upstream kernel to solve the problem.
Same problem here, fixed by selecting GRUB 2
Another report of this issue with latest Linode kernel. 30 machines went down when running apt update
Here's an alternative work-around, provided by Linode support (if you really want to keep using the latest kernel):
As root or sudo:
$ mkdir -p /etc/systemd/system/containerd.service.d/
$ cat << EOF > /etc/systemd/system/containerd.service.d/override.conf [Service]
ExecStartPre=
EOF
$ systemctl restart docker
Linode kernel 4.18.16-x86_64-linode118
is missing overlay
module.
So I found this thread.
Right now no time to check if it is already reported in an ticket.
Thing I want to report here is that the above workaround, reported by @k_sze, where containerd.service
is overwritten to not do /sbin/modprobe overlay
works for us.
Thanks.
Docker 18.9.0 fix for latest Linode Kernel
Last week, Docker released a new version (18.9.0) that has been steadily failing against our latest Kernel. Our team spent a few days testing Docker 18.9.0 with Ubuntu 16.04 - 18.04 and Debian 9 in order to assist our customers with the best solution possible.
We found we could recreate the issue after installing Docker 18.9.0 according to the documentation.
Output from a fresh installation of Docker-CE on Ubuntu 16.04 includes the following error messages:
Job for containerd.service failed because the control process exited with error code.
See "systemctl status containerd.service" and "journalctl -xe" for details.
containerd.service couldn't start.
...
A dependency job for docker.service failed. See 'journalctl -xe' for details.
docker.service couldn't start.
The code above shows us that the containerd service is looking for the OverlayFS module and failing, as support for OverlayFS is compiled into our kernel, and has been since version 113. As a result we believe this to be a bug introduced by the upstream docker package maintainers.
In addition to the workaround to switch to the GRUB2 kernel, we found the following solution in a Github Docker issue discussion which doesnt require switching kernels.
To apply this fix, please follow the below steps:
- Switch to root user
su -
- Run the following commands (these have been updated from the commands provided by @k_sze):
mkdir -p /etc/systemd/system/containerd.service.d/
cat << EOF > /etc/systemd/system/containerd.service.d/override.conf
[Service]
ExecStartPre=
EOF
systemctl daemon-reload
systemctl start docker
When we began investigating this most recent Docker issue, we received many questions asking whether our kernels still do not have OverlayFS enabled. While this was an issue in the past, OverlayFS has since been enabled in our latest kernels. This is confirmed by the following outputs:
$ uname -r
4.18.16-x86_64-linode118
$ grep -i overlay /proc/filesystems
nodev overlay
$ zcat /proc/config.gz | grep -i overlay
CONFIG_OF_OVERLAY=y
CONFIG_OVERLAY_FS=y
CONFIG_OVERLAY_FS_REDIRECT_DIR=y
# CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW is not set
CONFIG_OVERLAY_FS_INDEX=y
# CONFIG_OVERLAY_FS_NFS_EXPORT is not set
CONFIG_OVERLAY_FS_XINO_AUTO=y