Is there a best practice for (bash) scripting LKE cluster to deployment?

I'm (bash) scripting LKE cluster creation and the Deployment of a container that requires imageSecrets and a PVC.

I'm challenged by:

  1. Best signal for successful (ready) cluster creation.
  2. After Namespace creation, I occasionally get errors creating a Pod
  3. After PVC deletion, the Linode Volume does not appear to auto-delete

lke cluster-create and lke cluster-list do not appear to include a return value that can be used as a signal for the cluster's state:

[
  {
    "id": 12345,
    "created": "2023-02-14T00:00:00",
    "updated": "2023-02-14T00:00:00",
    "label": "foo",
    "region": "us-west",
    "k8s_version": "1.25",
    "control_plane": {
      "high_availability": false
    },
    "tags": []
  }
]

It would be useful if getting clusters included some value as to the server's state.

Instead, I'm using lke kubeconfig-view ${ID} --json and until the returned JSON includes kubeconfig rather than [{"field":"","reason":"Cluster kubeconfig is not yet available"}]

Is this the best way to do this?

NOTE Too small gaps between lke cluster-create and lke cluster-delete can delete the cluster without deleting underlying Linodes.

When I'm done, I delete everything (including the PVC). It's defined with:

storageClassName: linode-block-storage

But, the Linode Volume does not get deleted even after the Linode that it's bound to is deleted with the cluster's deletion. I'm having to enumerate the volumes as a fail-safe and delete any that remain.

NOTE Linode NodeBalancers are correctly programmed (creation|deletion) as Kubernetes Services type: LoadBalancer are apply|delete.

Scripts

Constants

IMAGE_LINODE="docker.io/linode/cli:5.31.0'

Create Cluster & KUBECONFIG

function create_linode_cluster () {
  local IMAGE="${IMAGE_LINODE}"
  local LABEL="foo"

  # Get latest Kubernetes version
  local VERSION=$(\
    podman run \
    --interactive --tty --rm \
    --volume=${HOME}/.config/linode-cli:/home/cli/.config/linode-cli \
    ${IMAGE} \
      lke versions-list --json \
    | jq -r '.[0].id') && echo ${VERSION}

  if [ -z "${VERSION}" ]
  then
    echo "Unable to determine Kubernetes version: ${VERSION}"
    return 1
  fi

  local ID=$(\
    podman run \
    --interactive --rm \
    --volume=${HOME}/.config/linode-cli:/home/cli/.config/linode-cli \
    ${IMAGE} \
      lke cluster-create \
      --label=${LABEL} \
      --region=us-west \
      --k8s_version=${VERSION} \
      --node_pools.type=g6-standard-1 \
      --node_pools.count=1 \
      --no-defaults \
      --json \
    | jq -r '.[0].id') && echo ${ID} 

  # Export LKE KUBECONFIG to temp file
  local CONFIG="${PWD}/tmp/linode.$(date +%y%m%d).yaml"

  local KUBECONFIG=""

  until [ -n "${KUBECONFIG}" ]
  do
    sleep 15s
    KUBECONFIG=$(\
      podman run \
      --interactive --rm \
      --volume=${HOME}/.config/linode-cli:/home/cli/.config/linode-cli \
      ${IMAGE} \
        lke kubeconfig-view ${ID} --json \
      | jq -r .[0].kubeconfig)
  done

  echo ${KUBECONFIG} | base64 --decode > ${CONFIG}
}

Create PVC, Secret, Deployment

function create_linode_foo_server () {

  local NAME="foo"
  local HOST="foo.example.com"
  local NAMESPACE="bar"
  local IMAGE=${IMAGE_FOO}

  local CONFIG="${PWD}/tmp/linode.$(date +%y%m%d).yaml"

  kubectl \
  create namespace ${NAMESPACE} \
  --kubeconfig=${CONFIG}

  # Create PVC
  kubectl apply \
  --filename=${PWD}/linode/kubernetes/pvc.yaml \
  --kubeconfig=${CONFIG} \
  --namespace=${NAMESPACE}

  # Create temporary Pod w/ PVC mount to X509 cert
  kubectl apply \
  --filename=${PWD}/linode/kubernetes/shell.yaml \
  --kubeconfig=${CONFIG} \
  --namespace=${NAMESPACE}

  # Wait for it to come Ready
  kubectl wait pod/shell \
  --for=condition=Ready \
  --namespace=${NAMESPACE} \
  --kubeconfig=${CONFIG}

  # Copy X509 cert and account into PVC
  kubectl cp \
    ${PWD}/linode/certs/. \
    ${NAMESPACE}/shell:/certs \
  --kubeconfig=${CONFIG}

  # Delete temporary Pod only
  kubectl delete \
  --filename=${PWD}/linode/kubernetes/shell.yaml \
  --kubeconfig=${CONFIG} \
  --namespace=${NAMESPACE}

  # Required because IMAGE_FOO is private
  kubectl create secret docker-registry ghcr \
  --kubeconfig=${CONFIG} \
  --namespace=${NAMESPACE} \
  --docker-server=reg.io \
  --docker-username=${USER} \
  --docker-password=${PASS} \
  --docker-email=${EMAIL}

  local DEPLOYMENT="${PWD}/tmp/linode.${NAME}.$(date +%y%m%d).yaml"

  sed \
  --expression="s|image: IMAGE|image: ${IMAGE}|g" \
  --expression="s|host=HOST|host=${HOST}|g" \
  ${PWD}/linode/kubernetes/autocert.tmpl > ${DEPLOYMENT}

  # Deploy autocert server w/ PVC mount to access ${HOST} X509 cert
  kubectl apply --filename=${DEPLOYMENT} \
  --kubeconfig=${CONFIG} \
  --namespace=${NAMESPACE}
}

1 Reply

You may find that Terraform and the Linode Terraform Provider offer some of the functionality you're looking for with a bit more ease of use. Looking over the Deploy a Linode Kubernetes Engine Cluster Using Terraform documentation, a success message is printed to your output after the cluster is created with Terraform.

You also mention that your PVC is not being deleted even though it is defined as storageClassName: linode-block-storage. Again, according to the documentation, this should allow for the PVC to be deleted when the cluster is deleted. If this is not occurring during your deletion process, I suggest doublechecking what you have named above as pvc.yaml file to ensure the configs are correct. You may also want to try to delete the namespace where your PVC resides first, which will delete the PVC and the pod it is attached to, then delete the cluster. If this doesn't resolve your issue, I suggest bringing this to the attention of the Support Team in a ticket.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct