Can I use an Alternative CNI with LKE?
Simply put, I am not a fan of Calico for many reasons. I'd like to replace this CNI with another, but I'm not sure how much effort this will be and if it's even feasible on LKE due to the way master and node communication is setup.
Observations
- It appears the Calico on LKE is very old:
3.9.2(Oct 2019) - Control plane routes traffic to ClusterIP services
- I can see calico IP to IP mesh among nodes, but it does not appear the mesh extends to master.
- I don't see any additional BGP peers defined in k8s CRD (bgppeers.crd.projectcalico.org) so it appears BGP is strictly between the nodes. Unless there's another configuration I'm missing.
Questions
- Is Calico used on LKE master control plane?
- Can the LKE control plane work with another CNI?
- How does the master to ClusterIP pod communication work from a network view? Is BGP even relevant or is there some other thing going on? Perhaps some non-Calico magic?
Consider Cilium CNI
To cut to the chase, I'm looking for Cilium support. I would really suggest taking a look it.
I've used Cilium in production on Kubernetes for years and it has the right set of features for securing Kubernetes networking, and they are laser-focused on building a modern stack using eBPF instead of iptables.
They have some killer features that should help to contain the complexity of Kubernetes networking and security within Cilium: removing kube-proxy, node-local services, node firewall and DNS-based network policies.
Beyond the tech, I think they have the better team behind it, that's why I'm still onboard the Cilium train. DigitalOcean clusters come with Cilium as well.
If I can get some answers about the master to clusterIP communication, I can experiment with other CNIs myself. I just don't want to go down a path that is doomed to fail.
Thanks!
3 Replies
thorner
Linode Staff
Hi there!
Firstly, thanks for being so diligent in writing up your inquiry! For security reasons there are some details regarding our infrastructure that I am unable to disclose, however I'd love to answer your questions and I'll be happy to provide as much info as I can.
- Is Calico used on LKE master control plane?
This isn't something I am able to disclose, so there is not much info I can provide for this first question, I'm afraid.
- Can the LKE control plane work with another CNI?
Currently, LKE only supports a single CNI (Calico), however, I can confirm that our team plans to add support for Cilium in the future. I can't give an exact ETA on that, but it's on our radar and is a planned feature for LKE.
- How does the master to ClusterIP pod communication work from a network view? Is BGP even relevant or is there some other thing going on? Perhaps some non-Calico magic?
Master to ClusterIP and pod network communication is accomplished via encrypted tunnels established and configured with Wireguard from the managed control plane to each LKE node.
It seems like your end goal here is to switch your CNI from Calico to Cilium. This is an unsupported configuration, and any modifications you make to the LKE deployed resources in your cluster (I.e. Calico, linode-csi, etc.) will be overwritten on the following LKE release cycle, which is about every 2 weeks.
I know this is probably not the answer you were looking for, but once we support Cilium we will be sure to make a fuss about it to let everyone know :)
Hope that helps! Let me know if you have any other questions.
Tim H.
Linode Kubernetes Team
Hi @thorner,
I'm really happy you are working on Cilium support, thanks for your detailed response. Keep up the great work!
Kekoa
