Questions: "Security Notice: Linode Manager Password Reset"
sent from Linode and confirmed on the blog
> Dear Linode customer,
Linode administrators have discovered and blocked suspicious activity on the Linode network. This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed.
We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems. We have implemented all appropriate measures to provide the maximum amount of protection to our customers. Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset. In so doing, we have immediately expired all current passwords. You will be prompted to create a new password the next time that you log into the Linode Manager. We also recommend changing your LISH passwords and, if applicable, regenerating your API key.
Would it also be suggested to change SSH keys, or is that an unnecessary step? What other security measures are all of you making, such as changing login credentials? It's hard for me to figure out what needs to be done given the lack of details….
14 Replies
Hopefully there will be some more clarification in the near future.
@Ghan_04:
Linode's biggest weakness is probably details regarding announcements like this. That being said, a "weakness" for Linode is still pretty damn impressive compared to most providers.
Hopefully there will be some more clarification in the near future.
@yaz:
Would it also be suggested to change SSH keys, or is that an unnecessary step? What other security measures are all of you making, such as changing login credentials? It's hard for me to figure out what needs to be done given the lack of details….
Given the password reset, I'm focusing on access points to Linode Manager functionality. The password change is enforced, but it seems to me all API keys should be regenerated - I'm not sure what "if applicable" means, since even if you aren't currently using them, they appear to always be valid, so can be used for access. Plus the LISH SSH password, as mentioned, if you've assigned one. I think that's about it.
As far as SSH, if you mean the public keys you've installed for use in accessing LISH, no, I can't see why you'd need to change those, since, well, they're public. In and of themselves they aren't of any use to gain access, since they're only paired with your private key which Linode servers never see. You only need to worry about changing those if your private key is compromised.
That is, unless there's a concern that you'd be fraudulently granted access to some hackers machine :-)
– David
@db3l:
As far as SSH, if you mean the public keys you've installed for use in accessing LISH, no, I can't see why you'd need to change those, since, well, they're public. In and of themselves they aren't of any use to gain access, since they're only paired with your private key which Linode servers never see. You only need to worry about changing those if your private key is compromised.
I was more wondering if someone theoretically got access to our Linode server–if things were compromised enough they could use my login to go through Linode Manager (+/- despite two factor authentication) and login via LISH--and obtained a copy of an SSH-RSA key used to login, shouldn't that be changed? Because even if the key is protected by a passphrase, that could eventually be cracked and used to gain access to the Linode. I think this makes sense; apologies if it doesn't.
@yaz:
(…) and obtained a copy of an SSH-RSA key used to login (…)
I'm not quite certain which ssh key you are talking about here? The one you have configured for LISH access, or some other key stored locally on your Linode - e.g., in the guest filesystem? For example, when you generate a key pair, you'll typically have two files (for example, "idrsa" (private) and "idrsa.pub" (public)). The private key must be kept safe as it's the key to your authentication and allows you to identify yourself to anyone you've given the public key to. The public key (which is typically the one with an "ssh-rsa" prefix in the file) is designed to be published anywhere you want to be able to assert your identity.
If the question is about your public key, there's no risk of exposure - those are expected to be exposed. However, if you're talking about a compromise of the private key, then sure, if you believe your guest itself and its filesystem data may have been compromised (which for me at least the Linode announcement isn't strong enough to encourage believing to be a likely possibility) and you have private keys there, you may wish to re-generate the key pairs. And then of course update anyone who had your old public key.
If I understand your scenario correctly it involves getting into the Linode Manager (which doesn't have two factor but perhaps you meant a failure of whitelisting?), then using LISH to access your console, cracking an account on your Linode to log in(*), and from there compromising your locally stored private keys. Theoretically possible? Sure. But probably unlikely, at least at this point.
If it were me (well, I guess it also is in this case), and again barring other evidence, I wouldn't worry too much about that scenario. With that said, it's not like regenerating the keys is a hard task (depending on how many places you may need to then update the new public key) so it's not like doing so is harmful either.
– David
(*) This assumes you don't leave yourself logged in on the console, in which case LISH access would be sufficient to have access to your guest, something I don't recommend.
@db3l:
If the question is about your public key, there's no risk of exposure - those are expected to be exposed. However, if you're talking about a compromise of the private key, then sure, if you believe your guest itself and its filesystem data may have been compromised (which for me at least the Linode announcement isn't strong enough to encourage believing to be a likely possibility) and you have private keys there, you may wish to re-generate the key pairs. And then of course update anyone who had your old public key.
If I understand your scenario correctly it involves getting into the Linode Manager (which doesn't have two factor but perhaps you meant a failure of whitelisting?), then using LISH to access your console, cracking an account on your Linode to log in(*), and from there compromising your locally stored private keys. Theoretically possible? Sure. But probably unlikely, at least at this point.
You nailed it, and sorry for my confusion which undoubtedly comes from being an amateur and not having a good enough understanding of these things. First, yes, I meant the IP whitelisting function of Linode Manager, and I shouldn't have used the term "two-factor." For the sake of discussion, host/server=Linode and local/client=my local computer which I use to login via SSH to the host. The potentially compromised key I was referring to indeed is the public key that's stored on the Linode under /home/user/.ssh/authorized_keys. I forgot that the private, passphrase-protected key on my local computer that pairs up with what's on the Linode is different from the public key.
I have a basic understanding of how key pairs can be used to encrypt email by giving other people your public key while you retain the private key. The email example makes it easy for me to understand how it's ok to give people a public key. However, my confusion comes from conceptually not understanding how a login is handled with the server having a public key and the client having the private key. Therefore, I didn't know if someone gaining access to this public key stored on the Linode–which I don't believe is routinely shared by the server--was risky.
I appreciate your patience, and I hope I've gotten things right in terms of the terminology. If not, I'm sorry; I'm going to find some reading materials about my question regarding how logins are handled with the key pair right now.
Edit:
For anyone who's still learning like me, I think this explanation from ArchWiki
> When an SSH server has your public key on file and sees you requesting a connection, it uses your public key to construct and send you a challenge. This challenge is like a coded message and it must be met with the appropriate response before the server will grant you access. What makes this coded message particularly secure is that it can only be understood by someone with the private key. While the public key can be used to encrypt the message, it cannot be used to decrypt that very same message. Only you, the holder of the private key, will be able to correctly understand the challenge and produce the correct response.
@trisager:
Would be VERY nice if the simple username/password login to the Linode Manager was improved on.
Yubikeys would do the job. They are cheap enough and could be added as an optional extra security measure.
They would only improve one aspect of the security though, all the others remain unknowns.
Is that OK or should I buy a hardware random number generator and make them all 200 chars long?
@sednet:
Yubikeys would do the job. They are cheap enough and could be added as an optional extra security measure.
They would only improve one aspect of the security though, all the others remain unknowns.
True. But the implication of the security notice is that there is a risk - however small - that login credentials stored with Linode could have leaked. A Yubikey, Google Authenticator, or something else like that would make a leak less of an issue.
@trisager:
@sednet:Yubikeys would do the job. They are cheap enough and could be added as an optional extra security measure.
They would only improve one aspect of the security though, all the others remain unknowns.
True. But the implication of the security notice is that there is a risk - however small - that login credentials stored with Linode could have leaked. A Yubikey, Google Authenticator, or something else like that would make a leak less of an issue.
Sure. I didn't mean to imply that using Yubikey's or something like that wasn't a good idea.
What worries me is that we don't know what happened here and if the last incident of this nature is anything to go by we never will. I can't make decisions on unknowns, I can only assume the worst.
Although they did not say it directly… and this is just speculation… here is what I feel had happened…
A customers site was hacked (for this reaction must be a big site or a customer with lots of linodes).
No indications of how the hackers got on. The sites may have had very good security wrt regular practices/patches/firewalls/etc; or was linode managed so they were aware of its configuration/patches security wise.
There were logins using lish or to the manager that were from the ip of the customer, but the customer could not account nor confirm if they were legitimate logins.
Since one of the possible paths was through lish/manager this is where the "abundance of caution" item comes into play.
Its one of those security items… if you cannot find the cause, then suspect everything is compromised no matter how unlikely that it could be the reason.
Using a third party mailer out of the sudden to send the password reset messages wasn't exactly an intelligent move to make either.