got `lost connection to pod` with `kubectl port-forward`
I need to connect directly to the svcs in staging namespace in the development process.
So I use kubectl port-forward --namespace staging svc/xxxx 1234:1234
Then I found that it would prompt lost connection to pod every 1 minute or so. Time is not fixed. This is true even if the connection is not idle.
Now i can only use
while true; do kubectl port-forward --namespace staging svc/xxxx 1234:1234; done
I have actively investigated this matter. Some people say it is because of the configuration problem of streamingConnectionIdleTimeout. Then I checked this value, it is 4h0m0s.
So I think this should be a limitation of linode?
This affects my development process, because I always encounter errors like EOF.
9 Replies
jyoo
Linode Staff
This is related to an issue on our side. Our Linode Kubernetes Engine team has a good understanding of why you're encountering this timeout limit, and they're working on a solution so that you don't see frequent timeouts moving forward. We'll make an announcement once this issue has been resolved, so keep an eye on our blog for updates.
Any update of this issue? It's extremely frustrating :(
bbigger
Linode Staff
We're still looking into adjusting the timeout limit — the consideration is a bit involved, though I can attest to how frustrating it is to have kubectl port-forward connections dropped after a brief period of inactivity.
In the meantime, you can keep the connection alive by using netcat (nc) in a while loop instead. Here's the sequence that I ended up using (borrowing from @smile's example):
- Run
kubectl port-forward:
kubectl port-forward --namespace staging svc/xxxx 1234:1234
- In a separate session, run
ncto the port every 10 seconds:
while true ; do nc -vz 127.0.0.1 1234 ; sleep 10 ; done
The regular "pings" will keep the connection from timing out, so you should never see a "lost connection to pod" or related EOF error.
I found another situation.
The solution may not be feasible for some services.
eg: postgresql
bbigger
Linode Staff
@smile Indeed, this approach may not work for every service. Although, for postgresql, the pg_isready utility can be used in the same way to keep a kubectl port-forward to a postgresql service from timing out. For example:
kubectl port-forward svc/postgresql 5432:5432 &
while true ; do pg_isready -d postgres -h 127.0.0.1 -p 5432 -U postgres ; sleep 10 ; done
(This will run the kubectl port-forward command in the background.)
You are simply amazing. Just solved my problem perfectly, and let me learn knowledge. thank you very much. respect!
@bbigger Is there any updates on this issue?
bbigger
Linode Staff
@man-from-geelong Yes! We've extended the timeout limit to 1 hour.
However, if you're still seeing lost connections before 1 hour, a timeout outside of Linode's systems (such as one caused by an intermediate VPN configuration) may be the cause.
If you suspect that lost connections are caused by an issue on our end, the best route would be to open a Support ticket so that we can investigate.