Mounting an existing volume to LKE

@JonathanKurumi the Linode CSI driver doesn't currently support remounting existing disk. At this time, you'll need to move your data over onto the new volumes. I understand how inconvenient this is, and I've made sure our developers are aware of this feature request. I'm also going to create a ticket in our internal tracker to record this feature request.

Existing volumes can be mounted, since that is an inherent property of the CSI driver. For instance, if you backup a cluster with CSI devices, and restore that cluster, your CSI devices (and existing block storage devices) would have to be remounted.

NOTE (2023-10-28): LKE's Linode CSI runs with a service account that only has permission to storage that it created.

The trick is in discovering the correct incantation of volume ids and labels to embed in the PV yaml.

https://github.com/linode/linode-blockstorage-csi-driver/issues/24 is an open issue looking for improvements in the CSI documentation for the Static Provisioning workflow.

The PV and PVC yaml below work for me. Replace {ID}-{LABEL} with your Volume ID and Label (which you can get from linode-cli). Update the storage size too, I used the default minimum size of 10Gi. Volumes must reside in the same region as the cluster and the volume must not already be attached to a Linode instance.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: existing-volume
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  csi:
    driver: linodebs.csi.linode.com
    fsType: ext4
    volumeHandle: {ID}-{Label}
  persistentVolumeReclaimPolicy: Retain
  storageClassName: linode-block-storage-retain
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name:  my-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: linode-block-storage-retain
  volumeMode: Filesystem
  volumeName: existing-volume
---

You can test this by creating the following busybox pod:

kind: Pod
apiVersion: v1
metadata:
  name: busybox-pod
spec:
  containers:
    - name: busybox-container
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: data-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: my-claim

Attach to the pod to explore your data:

kubectl exec -it busybox-pod -- ls -la /data # /bin/sh is available

I've tried with the same yaml file, updating {ID}-{Label} with my volume information from linode-cli.

The volume is not attached to any linode.

I've the following error in the pod events when launching it:

Warning FailedAttachVolume 6s (x5 over 14s) attachdetach-controller AttachVolume.Attach failed for volume "existing-volume" : rpc error: code = Internal desc = [403] Unauthorized

What's wrong?

The presence of an 403 Unauthorized error makes me believe that you might not be passing along a valid Kubernetes secret through the Linode Block Storage CSI driver.

This Kubernetes secret is used to store a Linode Personal Access Token (or PAT for short). The CSI driver must have a valid PAT so that it has the authorization to modify your Linode account's services.

You may generate and review your PATs review on this page in Cloud Manager:

• https://cloud.linode.com/profile/tokens

I would ensure that your token has read/write access to Kubernetes and Block Storage at the very least. You may need more access permissions depending on what you're trying to achieve.

Once you generate this token, you should be able to supply it as a Kubernetes secret with these instructions in our CSI driver's README.

I hope my suggestion here resolves this issue for you. If you're still having trouble, we'll be happy to assist you further.

@jo_pis3 have you solve that problem??

Warning FailedAttachVolume 6s (x5 over 14s) attachdetach-controller AttachVolume.Attach failed for volume "existing-volume" : rpc error: code = Internal desc = [403] Unauthorized

I follow any instruction here but it's not working. by the way I make volume manually from admin volume manager, is it wrong? any body help me? thanks.

Worked for me after upgrade to NVMe sotrage in Frankfurt region.

I'm trying my best to re-attach storage in order to copy the data across, but it's kind of chicken and egg situation, you can't copy until to attach, all I can do is attach new storage devices but not attach the old one to copy the data across. And if you were able to attach and copy, it kind of defeats the point right?

Any help or guidelines/documentation would be helpful, I've been going at this for a few hours with out much luck. I've been using multiple cloud providers and they appear not to have the same issue (they use selector, match labels - https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector) maybe something can be done with the linode csi driver to catch up maybe?

Many thanks team.

Since this question was asked, we've gotten some clarity from our LKE devs on the issue. While the linode-blockstorage-csi-driver can support mounting existing volumes as mentioned by @Marques, this is likely on a self-managed K8s cluster and not on an LKE cluster.

The reason you cannot attach pre-existing volumes to an LKE cluster is because of the linode secret. When an LKE cluster is created, the linode secret is tied to a hidden service user and the API key is scoped to only allow access to the resources created by that service user. Since the Block Storage Volume you are trying to mount was not created by that particular service user tied to your LKE cluster, it shares no relationship with the API key and therefore, you receive the 403 errors.

As a work around, you can create a PVC and, along with your existing volume, mount it to a new instance that is not part of your LKE cluster. From there, you can copy the data from the original volume to the PVC. Next, detach your PVC from the Linode, and mount it to your pod as you normally would using kubectl. From there, you should be good to go.

@byteloch - We appreciate the feedback. I have taken your request for this feature and passed it along to the team. If it gets implemented in the future, we'll be sure to follow up with you.

Getting tired of these Volume issues to be honest.

My Postgres app redeployed and was assigned a new node. The volume got automatically detached and attached to that new node. Container/Pod failed to start with Mounting errors. I redeployed the Pod back onto it's original Node. Volume was successfully mounted back to the old Node, but the Pod/Container still won't mount:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 30m (x9 over 50m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-n6t5h]: timed out waiting for the condition
Warning FailedMount 16m (x2 over 25m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-n6t5h dshm]: timed out waiting for the condition
Warning FailedMount 12m (x3 over 39m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-n6t5h dshm data]: timed out waiting for the condition
Warning FailedMount 92s (x33 over 52m) kubelet MountVolume.MountDevice failed for volume "pvc-19d050b1a14040c6" : rpc error: code = Internal desc = Unable to find device path out of attempted paths: [/dev/disk/by-id/linode-pvc19d050b1a14040c6 /dev/disk/by-id/scsi-0Linode_Volume_pvc19d050b1a14040c6]<

PVC description

Name: data-fanzy-postgresql-dev-0
Namespace: fanzy-dev
StorageClass: linode-block-storage-retain
Status: Bound
Volume: pvc-19d050b1a14040c6
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=fanzy-postgresql-dev
app.kubernetes.io/name=postgresql
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: linodebs.csi.linode.com
volume.kubernetes.io/storage-provisioner: linodebs.csi.linode.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: fanzy-postgresql-dev-0
Events: <none></none>

PV description

Name: pvc-19d050b1a14040c6
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: linodebs.csi.linode.com
Finalizers: [kubernetes.io/pv-protection external-attacher/linodebs-csi-linode-com]
StorageClass: linode-block-storage-retain
Status: Bound
Claim: fanzy-dev/data-fanzy-postgresql-dev-0
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: linodebs.csi.linode.com
FSType: ext4
VolumeHandle: 516140-pvc19d050b1a14040c6
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1662712251649-8081-linodebs.csi.linode.com
Events: <none></none></none></none>

Now I've noticed that even newly created PVC are failing to get attached to new pods/containers with the same error.

I ran through this example (https://github.com/linode/linode-blockstorage-csi-driver#create-a-kubernetes-secret) and reinstalled the drivers. PVC gets successfully created but fails to mount.

kubectl get pvc/csi-example-pvc pods/csi-example-pod

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/csi-example-pvc Bound pvc-c0ea8df9e5684244 10Gi RWO linode-block-storage-retain 21m

NAME READY STATUS RESTARTS AGE
pod/csi-example-pod 0/1 ContainerCreating 0 21m

Here's the error description from the pod:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned default/csi-example-pod to lke71838-112699-635487f6efa8
Warning FailedMount 3m38s kubelet Unable to attach or mount volumes: unmounted volumes=[csi-example-volume], unattached volumes=[kube-api-access-zvksd csi-example-volume]: timed out waiting for the condition
Warning FailedMount 83s (x5 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[csi-example-volume], unattached volumes=[csi-example-volume kube-api-access-zvksd]: timed out waiting for the condition
Warning FailedAttachVolume 14s (x7 over 12m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-c0ea8df9e5684244" : Attach timeout for volume 802990-pvcc0ea8df9e5684244

I confirmed that LKE does not have access to the desired volumes, only volumes the cluster created itself. These are effectively the limitations of a Linode Restricted user account that is given permission to create volumes (and can subsequently manage those volumes). The LKE Service Account is a special kind of Linode Restricted User Account that account owners can not grant additional privileges to. Therefore, we can not give the cluster service account access to volumes it did not create.

The instructions above can be used to mount a CSI volume clone or a previously created volume. This is helpful if you've removed the PV record and need to recreate it.

I'm going to revise my earlier comment since it is referenced elsewhere.

More details in https://github.com/linode/linode-blockstorage-csi-driver/issues/24#issuecomment-1783835385

I've also included a suggestion there for further consideration on the limitations of this approach. Migrating storage from one LKE cluster to another would benefit from features to extend grants on these service accounts or the CSI storage classes.

There are always workarounds.

This is still an ongoing issue for me, even with new volumes.

Either the "403 Unauthorised" or "Detected changes to resource volume which is currently being deleted. Volume Attach failed / volume attachment is being deleted". (Source: attachdetach-controller)

Of course, I have followed the instructions above, but it has not resolved the original issue.

How has everyone been living with this issue up till now?

So far the things the workaround that I have done are:

• Reboot/recycle the cluster/nodes over and over and over again until it randomly works. It takes at least 12 - 14 attempts, sometimes more before it starts working.
• If the first option does not work, it's very inconsistent in behaviour, then repeatedly clone and re-attach the storage 30 - 40 times before it starts working again.

I have these set up on a script so it's not too much effort at the time, but I dread everything there are changes required to the cluster that need to attach/re-attach storage.

It's so strange because when it does work, I can see a set of messages on the cloud dashboard informing me that the storage has been attached, so I know it has finally worked.

I've managed to get onto a new error pretty consistently each time:

MountVolume.MountDevice failed for volume "ecs-database-volume" : rpc error: code = Internal desc = Unable to find device path out of attempted paths: [/dev/disk/by-id/linode-ecs-database-volume /dev/disk/by-id/scsi-0Linode_Volume_ecs-database-volume]

Of course it is attached… so confused:

$ kubectl get volumeattachment
NAME                                                                   ATTACHER                  PV                         NODE                            ATTACHED   AGE
csi-e191e89bba9cef08e865165487e47b2ac6c6e96dbbc5cf848947d500209cb8ab   linodebs.csi.linode.com   ecs-database-volume        lke225001-325519-59ea0a3b0000   true       4m53s

My only fix is to keep rebooting until all the drives attach correctly. Sometimes I get 2 of the 4 mount, or 1 of the 4, and I just hope at some point all 4 mounts.