[TOP TIP] The Linode Private Network/IP is not private at all
Unfortunately, a private IP address gives EVERYONE on the same data centre access to your Linode server, not just your own servers.
The implications are very important. Many admins don't know this, thus they don't take extra steps to secure their servers, for example, a quick scan shows that many leave open access to SQL databases on port 3306. The same issue affected many recent hacks in Amazon's AWS services and unprotected S3 buckets, the admins failed to understand how accessible their servers are within the Amazon private network.
For the fun of it, I left open some HTTP/HTTPS ports to see what comes through the private network, here are some examples:
"HEAD / HTTP/1.1"
"GET /sftp-config.json HTTP/1.1"
"GET /wp-login.php HTTP/1.1"
While they seem simple scans to detect Wordpress or badly configured software, this is only the tip of the iceberg, since the scans for SQL databases is a lot more scary because they can quickly detect badly configured databases with no root password.
What is interesting, is that many of those scans also come via public internet interfaces, for example:
li276-166.members.linode.com - - [] "HEAD /wp-login.php HTTP/1.1"
li1447-246.members.linode.com - - [] "GET /sftp-config.json HTTP/1.1"
If you are using CentOS, it is easy to use firewalld to define rules for the private interface, the most secure thing to do, is to limit access based on IP address, something easy to maintain with Ansible.
21 Replies
If the OP is correct, this sounds like a serous security issue. Is there no answer to this person's clam?
Linode always recommended that you appropriately firewall private IP addresses because they are accessible by your neighbors.
Although, I just read through their current docs for adding private IPs and that doesn't seem to be mentioned anymore.
jcardillo
Linode Staff
@Jake - Private IPs in the same DC can communicate over the private network.
The purpose of private IPs is to connect Linodes you might have in the same DC, or to set up Linodes behind a NodeBalancer, for example.
Unfortunately, a private IP address gives EVERYONE on the same data centre access to your Linode server, not just your own servers.
To prevent this from occurring, you can setup firewall rules for your server. We have a few guides that can assist you with this option.
https://www.linode.com/docs/platform/manager/remote-access/#adding-private-ip-addresses
https://www.linode.com/docs/security/firewalls/control-network-traffic-with-iptables/
https://www.linode.com/docs/security/firewalls/configure-firewall-with-ufw/
https://www.linode.com/docs/security/firewalls/introduction-to-firewalld-on-centos/
jcardillo
Linode Staff
@BlueHound - Thanks for pointing that out. I've passed this on to our Docs team to see if this is something we can get added back in.
@jcardillo Just googled myself here, because I suspected that this was the case. The big question is, though: if I do SSL termination on the NodeBalancer, will the whole DC (or at least some of my neighbours) be able to listen in on the traffic between the NodeBalancer and my instances?
Also, from the documentation it wasn't clear (or I missed) that the private IP can be used with the NodeBalancer (but that's not a big deal).
jyoo
Linode Staff
As mentioned in the initial post, it is important to make sure your firewall is appropriately configured to protect against attacks on your private network. However, Linode filters traffic based on MAC and IP addresses, and Linode users cannot see other people's traffic. This includes your traffic between your NodeBalancer and your backend nodes. You might find our blog post about the private IP network to be helpful - there are some comments there that address this scenario :)
Hi!
Any modification about this topic?
I am try to connect to a vm with database service from another in the same dc, no fw rules. I can't access.
I try telnet to database and I found the error: No route to host.
I use nmap to try discovery open ports in my vm with database and only 22 is listen, but in local machine netsat say that is open.
Any help?
Thanks
Hi!
Any modification about this topic?
I am try to connect to a vm with database service from another in the same dc, no fw rules. I can't access.
I try telnet to database and I found the error: No route to host.
I use nmap to try discovery open ports in my vm with database and only 22 is listen, but in local machine netsat say that is open.
Any help?
Thanks
RESOLVED
You could use IPSec to secure the traffic on the private DC network address:
-- sw
P.S. You never assume any communication channel is private.
https://www.zerotier.com/ Works too, i have used it for years now.
While you can never assume, there is a difference between trusting Linode and trusting everyone on the same network.
Host-to-host solutions are hard to manage (at least without something like Consul).
Zerotier looks like a good fit, I'll see how independent it can be made from their infrastructure. (I saw you can install some component that helps with peer discovery, but as far as I can understand, even then you'll have to be able to reach their services. Which means if they go down, you probably can't join new instances to the network.)
FWIW, Linode will (hopefully this year) be starting a beta for private VLANs which, as I understand it, addresses this issue.
Although as @stevewi said, you should still not assume this is truly private and put safeguards in place.
I believe this is akin to DigitalOcean’s newly launched VPC feature.
Wait so Linode actually allow customers to probe other customers' stuff for vulns? Is it not enough that like half of china is probing 24/7, now we have to worry about local attacks as well? :(
Curious what the status on the vlan solution. I'm a bit appalled this is possible at all.
So, a linode user can just port scan the entire 192.168.0.0/16 B block with no repercussions? How on earth is this acceptable?
And software filtering doesn't really pass any reasonable security test, yes it might work, but this would fail PCI even just by the fact it's possible.
There is a massive misunderstanding being repeated often, particularly started by @jcardillo
Adding host-based firewall is not a "prevention", it will not make your so-called private IP somehow magically actually-private, the IP address remain quasi-private (routes and connections remain possible by all hosts in the datacenter)
The implications of this means any that many DoS vectors stay open, it also means you can be fingerprinted, and it also means a missconfiguration means you fail-unsafe - not fail-safe.
That last one being fail-unsafe perpetuates the unsafe-by-default security posture.
Consider the entire cybersecurity field is based on a triad of principals; Integrity, Confidentiality, and Availability
Consider that customers and partners are most concerned with Confidentiality; and Linode has non inherently provided.
Consider Integrity is subject to the failure-mode, and Linode is a fail-unsafe design
And consider Availability is the primary concern of any developer/operations professional, Linode has no inherent truly-private hosts when using private IP addresses leaving them open to DoS when the only known defence is applied.
These violations of the CIA triad provides no confidence at all, particularly under scrutiny by auditors and any/all attestations that say they audited Linode and found inherent security is solid, are simply untrustworthy (likely paid off auditors) after reading this thread.
Will I keep using Linode? Absolutely!
It's by far the best platform for Linux and community, without any shadow of doubt, I love them for this.
But if you want private networking, typically you expect inherent security, but you cannot reasonably expect it on Linode.
There's a very BIG difference between private and secure… Privacy does not imply security.
No computer networking scenario is inherently secure…even those advertised as such. If you want security, you must build/maintain it.
-- sw
I'm sorry, but how is a private IP any less secure than a public one? The private IP should just be treated as a shortcut to resources inside the datacenter and not that it is automatically secure in anyway. A server with a public IP is just as vulnerable, if not more than one that has a private IP. One should practice the same security protocols as they do on the public internet with proper secure termination whether that be HTTPS/SSL/TLS or SSH. Use public WAN firewall rules. Use authentication. Don't treat it like your own household or business LAN.
There might actually be a few use cases where 3rd parties may want to communicate within a single data center to avoid data costs through public internet providers. Other than that, I think their VLAN feature has solved most use cases.
Yeah, this should be outlined in the docs and have an information bubble next to the option when you are about to enable it.
Just discovered this thread after seeing multiple blocked requests to port 22 from 192.168.128.0/17 CIDR that are not mine.
Luckily for me, I never run ssh on the standard port 22. I unfortunately had naively allowed access to a database port from any 192.168.128.0/17 on a node which is quite common.
@jyoo what I have found most puzzling is that traffic from these private IPs bypass the Linode firewall: Am explicitly blocking all requests to port 22 already.