Cert-manager: problem with webhook - i/o timeout
Hi,
I'm trying to install the cert-manager using this instructions
The error is:
"failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": dial tcp 10.128.37.173:443: i/o timeout" logger="cert-manager.controller"
I installed using this:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml
Everything was created successfully
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
All pods are running:
kubectl get svc,pods,endpoints -n cert-manager
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.128.194.202 <none> 9402/TCP 22m
service/cert-manager-webhook ClusterIP 10.128.37.173 <none> 443/TCP 22m</none></none>
NAME READY STATUS RESTARTS AGE
pod/cert-manager-5798486f6b-scwzn 1/1 Running 0 22m
pod/cert-manager-cainjector-7666685ff5-chhvs 1/1 Running 0 22m
pod/cert-manager-webhook-5f594df789-jtzm4 1/1 Running 0 22m
NAME ENDPOINTS AGE
endpoints/cert-manager 10.2.0.137:9402 22m
endpoints/cert-manager-webhook 10.2.0.138:10250 22m
But the API check it is not OK:
cmctl check api
I see this error:
Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I tried the troubleshooting and can't solve the problem.
Other pods can access the webhook, like this:
curl -vsS https://cert-manager-webhook.cert-manager.svc:443/validate --request POST --data '{"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1","request":{"requestKind":{"group":"cert-manager.io","version":"v1","kind":"Certificate"},"requestResource":{"group":"cert-manager.io","version":"v1","resource":"certificates"},"name":"foo","namespace":"default","operation":"CREATE","object":{"apiVersion":"cert-manager.io/v1","kind":"Certificate","spec":{"dnsNames":["foo"],"issuerRef":{"group":"cert-manager.io","kind":"Issuer","name":"letsencrypt"},"secretName":"foo","usages":["digital signature"]}}}}' -H "Content-Type: application/json" -k
Response
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< content-length: 135
< date: Mon, 01 Jul 2024 01:04:43 GMT
{"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1","response":{"uid":"","allowed":true,"status":{"metadata":{},"code":200}}}
I hope for anyone could help.
2 Replies
CassandraD
Linode Staff
I used the same instructions you shared and did not run into the same errors. You've also already shared the troubleshooting guide I pass along when customers run into similar issues, so I'm not sure how much this will help.
First, you may want to just try again. You can uninstall and delete anything related to cert-manager and go through the instructions again.
When you try again, you can look for any issues in your events and logs for the pods that are created. Because the events don't last very long, I think doing this again and checking for any issues while you're working could be helpful.
kubectl get events -A
kubectl logs $podname -n cert-manager
You could also check to see if there are any issues with the pods that control networking within your cluster. You can investigate kube-system pods by running kubectl get pods -A and then describing or getting logs for anything that may be having trouble. Usually there would be issues beyond one webhook if that were the case, but it seemed worth looking at.
Hopefully you've figured something out because I know it's been a few weeks, but I wanted to try to offer some suggestions. If you have additional information about what happens if you try again, hopefully someone else from our Community will have a more concrete answer for you.
I'm going through the same thing and it's been driving me crazy! I didn't have this issue on Digital Ocean, and this seems to be specifically a Linode problem.
Also here too: https://www.linode.com/community/questions/20613/lets-encrypt-on-kubernetes-with-cert-manager-webhook-failure-when-creating-clust