Use a specific nodebalancer id with ingress-nginx?
I want to select a pre-existing nodebalancer for ingress-nginx instead of creating one when I install the ingress-nginx chart.
The nodebalancer annotations reference for LKE indicates that I can do this:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id: "301323"
…but the external IP is always <pending>
for the ingress-nginx controller's LoadBalancer service. It's as if CCM isn't noticing the annotation ending "-nodebalancer-id", or there's some kind of mismatch between the configuration of the nodebalancer I created externally with Terraform and the requirements of the ingress-nginx controller service.
7 Replies
✓ Best Answer
I was able to work around this by discovering the public IP of the Nodebalancer through another resource.
data "kubernetes_service" "ingress_nginx_controller" {
depends_on = [helm_release.ingress_nginx]
metadata {
name = "ingress-nginx-controller"
namespace = "ingress-nginx"
}
}
And subsequently create the Linode domain record with this awkward reference.
resource "linode_domain_record" "wildcard_record" {
domain_id = linode_domain.cluster_zone.id
name = "*"
record_type = "A"
target = data.kubernetes_service.ingress_nginx_controller.status[0]["load_balancer"][0]["ingress"][0]["ip"]
ttl_sec = var.wildcard_ttl_sec
}
The reason I need to pre-create the nodebalancer is so that I can discover the IPv4 in the Terraform plan so I can use it to also create a DNS record.
The data resource for nodebalancers requires the numeric ID of the nodebalancer to lookup the IPv4, so I can't discover the IPv4 by label or some other criteria after it's created by the ingress-nginx chart.
Now I see these errors in the LoadBalancer service's describe view.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 9m (x9 over 24m) service-controller Ensuring load balancer
Warning SyncLoadBalancerFailed 9m (x9 over 24m) service-controller Error syncing load balancer: failed to ensure load balancer: [403] Unauthorized
Warning UpdateLoadBalancerFailed 4m20s (x13 over 24m) service-controller Error updating load balancer with new hosts map[lke92205-139651-63e7cc9f7da4:{} lke92205-139651-63e7cc9fa724:{}]: [403] Unauthorized
I also noticed that the LoadBalancer Service becomes stuck in this pending state. Uninstalling the ingress-nginx chart doesn't remove the Service, and a new chart release with the same name can't be installed while it's occupying the {release}-controller
name in the same namespace.
I worked around this by getting the external IP from the Kubernetes Service resource after installing ingress-nginx and waiting for the LoadBalancer to be provisioned instead of pre-creating the Nodebalancer in the TF plan.
I am also facing an issue with:
service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id:
BTW, I am creating Nodebalancer via Traefik's service as defined in the helm chart.
However, upon it's creation I want to use it by referencing via annotation.
BTW, I am using the below script to get IP.
traefik-get-loadbalancer-ip.sh
!/bin/bash
take the json input and extract the CONTEXT variable
eval "$(jq -r '@sh "CONTEXT=(.context)"')"
CMD="kubectl -n traefik get services traefik -o jsonpath={.status.loadBalancer.ingress[0].ip} --context ${CONTEXT}"
keep checking until the loadbalancer IP (EXTERNAL-IP) is assigned to the traefik service.
NODEBALANCER_IP=$(${CMD})
until [[ ${NODEBALANCER_IP} != "" ]]
do
sleep 5
NODEBALANCER_IP=$(${CMD})
done
return json with the nodebalancer ip
jq -n --arg nodebalancer_ip ${NODEBALANCER_IP} '{nodebalancer_ip: $nodebalancer_ip}'
And calling it via:
// nodebalancer_ip output
data "external" "nodebalancer_ip" {
program = ["/bin/bash", "${path.module}/files/traefik-get-loadbalancer-ip.sh"]
query = {
context = "${var.kubectx}"
}
depends_on = [null_resource.traefik_install_wait]
}
Can someone from Linode support look into the issue with:
service.beta.kubernetes.io/linode-loadbalancer-nodebalancer-id:
Facing the same issue. nodebalancer-id
annotation does not work. Removed the annotation and started using external-dns
to automatically setup DNS for the allocated external-ip
.