Auto-routing traffic to k8s nodes from non-k8s nodes
Ok, I have a service routing conundrum and have done research into potential solutions, but could use assistance knowing what's possible on Linode. Apologies if I misuse the terms; I'm not a network specialist and just getting up to speed.
The setup and goals:
- I have an LKE cluster running some ClusterIP services.
- There are other non-k8s nodes in the same datacenter + private network.
- I want the k8s services to be highly available to the external servers (but not the public internet; firewall's a separate issue), not depending on traffic through a single-point-of-failure node
- It's ok to proxy traffic through k8s proxies running as NodePorts, but I want to avoid hard-coding a single node IP in case it goes down or the pod(s) move to another node. The idea is to leverage k8s' scheduling & routing features to send traffic to the service wherever it's running.
- Basically, the non-k8s nodes need to continuously & seamlessly know which node to send traffic to for a given service. Probably desirable to have a single virtual IP per service, that the other hosts can reliably use (with friendly DNS) and not care which machine it's running on.
- (A LoadBalancer / NodeBalancer isn't a solution because this is for cluster-internal traffic only, isolated from public internet)
Option 1: BGP with Calico
This would be a very cool solution if possible: we already have Calico running in the cluster. I just can't figure out if Linode's network infrastructure makes this possible for customers.
- Calico supports BGP peering and can advertise service IPs including ExternalIPs.
- I'm thinking that if we grant a floating/virtual IP (something in
192.168.128.0/17) to theserviceExternalIPthen Calico will broadcast, "hey! all traffic to the virtual IP should go to one of my appropriate nodes." From there it'll be directed by the kube networking internals. - This very technical Linode guide suggests that we can run a BGP daemon that will advertise where a floating/elastic IP is pointing. The setup details are a bit hairy & I'd need help translating that to Calico and finding what BGP peer specs to plug in. Is this worth pursuing?
Option 2: ARPing with MetalLB
I like this a little less because it requires installing & configuring a new package.
- Here's a simple example of a MetalLB setup to claim a virtual IP (or IP range)
- Is it permitted to put out our own ARP advertisements on the network? I have the sense that Linode allows this if we request floating IPs from support -- then those IPs are usable for things like this.
- But in the Linode cloud manager's networking tab for my k8s nodes, the option to use one of my existing floating IPs returns an error, saying it's not allowed for that particular node. Is it even possible to assign virtual IPs to LKE nodes / do I need to ask support? Ideally I could get a handful of floating IPs that will be assignable to any current or future LKE nodes in the cluster.
Option 3: VRRP with keepalived
This is my least favorite because traffic will only go to the primary server at any given time, leaving the others underused.
- I've set up keepalived before, using a Linode floating IP for failover. It's ok but I don't love it. The configuration is ugly and feels a bit antiquated. :P
- There are some k8s+keepalived projects around to streamline deployment
- The same caveat about being unable to use floating IPs applies
What to do, what to do. Any inside knowledge that can help? Thanks for reading!
2 Replies
rdaniels
Linode Staff
@shoreland You've done quite a lot of research for this already, and I applaud you for it! While I'm not an expert in k8s networking configurations, I wanted to answer your questions related to Linode's infrastructure and to provide you with some alternative solutions.
You can assign a single private IP address to your Linodes in Cloud Manager using IP sharing. You'll need to reach out to Support via a ticket if you want to add multiple private addresses to your Linodes.
Note: This feature is only currently available in the following data centers: Fremont, CA Dallas, TX London, UK Frankfurt, DE Singapore, SG Tokyo, JP
The Kubernetes community may be able to provide you with some more specific insider knowledge about ClusterIP configurations that can help point you in the right direction.
It might also be helpful to reach out to our Cloud Infrastructure Consultants to see if they are able to help you achieve your goals. This team works on a per contract basis and may be able to configure your setup, for an additional fee.
Hi @shoreland, it's been 2 months and I'm really curious if you've managed to set up any of the three alternatives you've listed in your original question. Also, bump to any Linode staff in case you have some official pointers.
In my case there's a few internal-only HTTP services I'd like to reach, and I've figured I'd use a hostNetwork ingress for that, which will need to be secured otherwise. Thankfully, shoreland's other post about Calico's network policies pointed me in the right direction and saved me a dozen hours of headaches, so thank you!
In any case, I don't know BGP enough to even attempt the first option, and the second one using MetalLB is a bit of a question mark, as even if Linodes seem to be connected to a regular layer 2 network with functioning ARP, it's not guaranteed it works on Linode's network