How do I install NFS on LKE nodes?
I'm trying to use NFS inside my Kubernetes cluster to provide ReadWriteMany volumes. However, it appears that I can't mount NFS file systems due to kubelet failing to mount the NFS file system:
MountVolume.SetUp failed for volume "pvc-bf72bfa1-c36a-44df-93d5-d3360d6e30b4" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/82382ceb-d7d5-455d-8f11-344c5fa73fd6/volumes/kubernetes.io~nfs/pvc-bf72bfa1-c36a-44df-93d5-d3360d6e30b4 --scope -- mount -t nfs -o vers=3 10.128.85.187:/export/pvc-bf72bfa1-c36a-44df-93d5-d3360d6e30b4 /var/lib/kubelet/pods/82382ceb-d7d5-455d-8f11-344c5fa73fd6/volumes/kubernetes.io~nfs/pvc-bf72bfa1-c36a-44df-93d5-d3360d6e30b4 Output: Running scope as unit: run-rb2cd034bae6b4205b77fc90ea0d96910.scope mount: wrong fs type, bad option, bad superblock on 10.128.85.187:/export/pvc-bf72bfa1-c36a-44df-93d5-d3360d6e30b4, missing codepage or helper program, or other error (for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program) In some cases useful info is found in syslog - try dmesg | tail or so.
I shut down one node, reset its root password, booted it back up, SSH'd into it, and discovered that /sbin/mount.nfs does not exist on the node.
It looks like these nodes are running Debian, so I can probably get it working by performing apt install nfs-common manually on all of my nodes, but is there an easier way to do that (specifically one that I won't forget to do when adding new nodes!)?
3 Replies
@jyorke --
You write:
It looks like these nodes are running Debian, so I can probably get it working by performing apt install nfs-common manually on all of my nodes, but is there an easier way to do that (specifically one that I won't forget to do when adding new nodes!)?
When you say node, is this a Linode or some Kubernetes construction?
If it's a Linode, you can mount NFS filesystems at boot time. See:
Use the /etc/fstab method.
-- sw
The Kubernetes nodes are Linodes, yes, but they use an “unknown” image pre-set by Linode without any SSH keys added or a known root password. Ideally, one would never log into the nodes, as Kubernetes handles orchestrating workloads to them. Getting the mount command issued isn’t the problem, the problem is that /sbin/mount.nfs does not exist. The only potential workaround I’ve found is:
kubectl drainto remove the workload from a node.- Shut down the Linode.
- Reset the root password on the Linode.
- Boot up the Linode.
- Connect via ssh as root and run
apt install nfs-common kubectl uncordonthe node to resume scheduling workloads.- Repeat 1-6 on all nodes in the cluster.
My concern with this is having to do this manual setup on every node is time consuming and is very likely to cause a failure in the future due to me forgetting that they need to be done when the cluster is scaled up.
An update on this today!
To debug another issue, I needed to spin up new nodes and SSH into them, and I found that the newest image (with KubeProxy/Kubelet v1.17.9) comes with NFS pre-installed!
So all I had to do was recycle my node pool through the control panel to upgrade the node OSes and everything now works without any manual steps.