Migrating to NodeBalancer in TCP mode with Proxy Protocol
I'm looking into adding a NodeBalancer to my existing site. I've set up the NodeBalancer in TCP mode. My application needs to know the original IP address, so I would like to use Proxy Protocol. I've run a test and I can get this setup working, but when Proxy Protocol is enabled in nginx, anything not going through the NodeBalancer can't connect.
If and when I decide to go ahead with NodeBalancer, it seems like this would cause an outage of up to 24 hours as the DNS changes propagate, because my options would seem to be:
- Enable Proxy Protocol in nginx, breaking those who have not received the DNS changes.
- Leave Proxy Protocol off in nginx until the DNS changes propagate, breaking those who have received the DNS changes until I turn it on.
- Leave Proxy Protocol off in nginx and in the NodeBalancer - things work for everyone, but the app doesn't get the real IP address for those who received the DNS changes until I update the configuration.
Is there another alternative here?
2 Replies
rdaniels
Linode Staff
@donny-don I see you've put a lot of thought into this. The options you laid out are the only available ones, to the best of my knowledge, when using NodeBalancers. While DNS propagation does take between 24-48 hours, it often times may complete quicker than that. You can utilize a website like DNS Checker or DNSMap.io to check the global propagation process.
That being said, depending on the application that you're using, you may be able use one of the setups listed here. Then, you can configure IP sharing of the original IP address as a failover option between multiple Linodes. It will require some additional configuration, but this option may help keep your service from being down longer than you like.
Thanks for the information.
To help anyone else with the same issue, what I did was:
- Create a mode in my app where IP address is "less" required (e.g. IP-address-based rate limiting is disabled).
- Many days before the migration, set the DNS TTL to the lowest value (5 minutes).
- On migration day, switch the app to the special mode.
- Change the DNS to point to the load balancer (which uses TCP without Proxy Protocol)
- Monitor the traffic that's going through the load balancer (tail -10000 /var/log/access.log | grep -c 192.168). It was about 50% after 5 minutes, 90% after an hour, 95% after 2 hours (and most of the remainder was bot traffic). This can also be monitoring by the proportion of traffic on the Linode attributed to "Private".
- Once an acceptable level of traffic is going through the load balancer, enable Proxy Connect in the load balancer and in nginx.
- Disable the special mode in the app.
- Put DNS TTL back to the default value.