Spin up and close down clusters on a schedule

I have a script that I am using cronjob for, running around 1 hour per day, using GPU resources, so only want to pay for the 1 hour that I use - I can setup the cronjob kubectl, but doing that I pay for the cluster even when I am not using it.

Any way around it?

1 Reply

This isn't something I've done before, but looking through previous tickets, it sounds like this is something you could probably do, though I don't have any idea of what the best practices may be.

You can create scripts that leverage the Linode API to manage LKE Clusters which could include creating and deleting the clusters. Another option to manage a cluster that has the configurations you want upon creation would be to use something like Terraform to deploy your cluster. Setting a cron job up on your local computer should work if you have everything set up to use our API.

While automating a cluster to be created and deleted for an hour a day is likely pretty doable since you're already familiar with cron jobs, the next challenge would be automating getting your workload onto the new cluster each day. I know I've seen users do similar things with Helm though that seems really complicated for the purpose of running a script.

You'd also need to worry about kubectl getting the updated cluster information from the kubeconfig and updating the context for commands. Again, I'm sure that could be automated using these same resources.

All this said, if you're running a script for just an hour a day, you may be able to do this more easily with a single compute instance rather than an LKE cluster. Since many of the benefits of Kubernetes wouldn't apply to a cluster you want to delete and remake every day, I'd recommend creating a golden image of the server running the script. You can then use the Linode API to create a script on your home computer that deploys a Linode using that image and then tears it down after the script has been given time to run.

Also, keep in mind that if you're using our GPU resources, they are extremely limited at the moment and deleting a GPU server may mean you lose access to that resources indefinitely.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct