Nodes won't start with recent k8s changes/upgrade: plugin type="calico" failed
I'm getting the following error on several nodes and I don't know when this means or what to do to fix it.
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Can anyone help me with this?
2 Replies
hdiep
Linode Staff
It seems that the failed to create pod sandbox error is a common issue when attempting to create a pod in Kubernetes. This error can be caused by issues with networking, and it can also be caused by system resource limit configurations. I was able to locate this third-party documentation that explains in detail where this error may stem from and how to troubleshoot it:
I was also able to find another thread where this other user experienced the same error that may prove useful to you:
Regarding the other error code in your output, plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/, I've found a few resources that mentions a couple of reasons why this may be occurring. I've provided a few threads below that may help with debugging your error output:
- Calico upgrade from 2.6.10 to 3.3.1 failure · Issue #2841 · projectcalico/calico · GitHub
- stat /var/lib/calico/nodename: no such file or directory problem,please help. · Issue #1795 · projectcalico/calico · GitHub
- directory: check that the calico/node container is running and has mounted /var/lib/calico/ 2022-07-01 09:30:56.583 [ERROR][4033924] plugin.go 518: Final result of CNI DEL was an error. error=stat /var/lib/calico/nodename: no such file or directory: check that the c Final result of CNI ADD/DEL was an error. error=stat /var/lib/calico/nodename · Issue #6309 · projectcalico/calico · GitHub
- Pods not running using Calico - Kubernetes - KodeKloud - DevOps Learning Community
Hopefully, these resources may explain why they're occurring and point you in the right direction!
It turns out the IP addresses on my nodes changes, and I have ACL set up for my mongo instance.