LAMP Security HOW-TO, IP Access Hardening Question
Recently, some hosted wordpress CMS tree-structures were targetted by spammers. The Spammers found world-writable folders (such as those needed by caching plugins) and uploaded rogue .htaccess/.php files to use my (Ubuntu Linux 11.04) Linode as a proxy where from spam message links would arrive to then be redirected to other servers.
My question is, is there a practice/method or HOW-TO I can follow to harden my linode against unauthorized IPs writing to any world-writable directory?
Thank you for any guidance in this,
Faisal
15 Replies
Here's some tips I'm following to reduce security hazards
World-Writable Files
Anyone can modify world-writable file resulting into a security issue. Use the following command to find all world writable and sticky bits set files:
find /dir -xdev -type d ( -perm -0002 -a ! -perm -1000 ) -print
You need to investigate each reported file and either set correct user and group permission or remove it.
If above command produces any output, fix each reported directory /dir using the chmod command (be careful with the following command):
find /webroot -xdev -type d ( -perm -0002 -a ! -perm -1000 ) -print0| xargs -0 chmod +t
Noowner Files
Files not owned by any user or group can pose a security problem. Just find them with the following command which do not belong to a valid user and a valid group
find /dir -xdev ( -nouser -o -nogroup ) -print
Make sure to configure iptables to block everything except what you absolutely need to allow. When possible, limit access to your ports to a specific IP (e.g. if the IP address of your home connection is always the same, tell iptables to only accept ssh connections (port 22 by default) from your home home's IP).
Also, if you want to allow users to upload stuff, have WordPress limit uploads to registered users, and require email activation for anybody to register. Generally, spambots won't know how to click the link in the email, they are normally designed to only find sites that they can register and send out their spam (in this case, upload the spam to a server), and that's all they do, hence they won't check their email accounts for the activation because spammers are too lazy to program that into their spambots.
If that still doesn't work and you really want to allow users to upload stuff, you can require admin-activation, meaning that users can register, but an admin will have to activate all accounts before they can be used.
If allowing users to upload files isn't necessary, it's best to disable that altogether. If you do allow it, limit the types of files people can upload, and don't let them use php in their coments. You may also look into some plugins on the WordPress site for spambot prevention.
If you've already done all this, tarball the files that were uploaded and move them off-site in case they need to be examined for exploits. Check your logs. Check your main system config files AND the config for your web sites. If the config files you absolutely need to keep look okay, move those off site. Backup your database and move off site. Redeploy your Linode, install all software you need (including Wordpress), and restore your config files and database. While your Linode is redeploying and your software reinstalling, double check your config files to make sure they aren't altered from what you set. Make sure all software is updated to the latest stable version offered by your distro, and update your distro altogether if it's outdated. Don't rely on your distro software repositories for things like WordPress, it's best to get those from the official site.
EDIT: If you do backup and restore, it should be enough to just grab the WordPress config files, and not WordPress itself. From there you should be able to re-download a fresh copy of WordPress and drop the config file into the correct directory. However, I'm not 100% sure on this since I've never used WordPress, so double check on their site. This does work on most sites, though.
If you don't need to keep your current WordPress install and you can start fresh without hurting anything, this would be better since something in WordPress or it's database my have been altered to make future exploits easier. The same goes for the rest of your software, and anything else in your database.
Well I'm definitely taking an inventory of the tree structures and ensuring world-write is disabled for any unnecessary directories. None of my sites offer any ability for direct user uploads. Any upload requirements have typically been internal ones needed by some wordpress plugins (for example, one plugin takes hotlinked images and converts them into local ones and requires a world writable upload directory to manage this function).
What I noticed was that these spambots do a recursive directory scan for writable directories (presumably over port 80). The rogue files were actually uploaded via the www-data user/group (I don't have anon-ftp, or weak ssh, so I'm assuming also via 80).
What I think I'd really like is some sort of security mechanism – similar to the Linode Manager, which triggers an IP to get blacklisted via .htaccess as well as fire an email to me the moment an unauthorized IP addy decides to sneak an upload.
I'm guessing that somehow a firewall daemon will be required for monitoring the IP information, but am unsure about how to conduct the blacklisting of the unauthorized IP addresses attempting to write to a directory. My gut would be to write a shell script which checks for log files created by the firewall, extract any offending IP's and write them into an .htaccess deny tag.
I'll update my findings here as I look into this.
chown -R www-data:www-data /srv/www
(make sure the R is uppercase)
The "-R" will tell chown to change the user:group ownership to www-data:www-data of the /srv/www directory and everything it can find inside it, and anything inside any subdirectories inside /srv/www, and so on until it can't find anything else inside /srv/www . You need to pay attention when using "-R", though – if you type the wrong directory, you can throw off your entire system by assigning the wrong user and group and thus throw off anything expecting a certain user and group, and create a major security hazard by allowing the wrong process or user to change things they ought not to. "-R" will also work with chmod. Again, you have to pay attention. To secure off your site, you can do:
chmod -R 550 /srv/www
chmod 770 /srv/www/yoursite.com/html/upload
550 will tell chmod to allow read and execute permissions to the user and group, and no permissions to everybody else. If www-data is both the user and the group, and only your web server uses www-data, this is the most ideal situation. However, if you need to allow uploads to a certain directory (in my example, that's /srv/www/yoursite.com/html/upload), you would use 770, which allows read/write/execute to the user and group, and no permissions to everybody else.
You also need to check the documentation for WordPress (or anything else you serve up) to see if it needs a certain set of permissions on certain files. Wrong permissions on anything critical to the site, such as config files, will cause most CMSes to malfunction.
If you're wandering about the numbers, read, write, and execute have a certain number assigned to them, and in order to grant a certain set of permissions using chmod, you can add those numbers together as needed. The numbers for the individual permissions are:
* Read = 4
Write = 2
Execute = 1</list>
So a '5' would mean Read and Execute since Read + Execute (4 + 1) = 5. Also keep the three digits in order: The first digit will be for the user permissions, the second digit for group permissions, and the third for others, so a 764 would allow the user all three permissions, group would have read and write, and others would have read-only.
There is another method to using chmod. You can also use u, g, and o to represent user, group, and others (or use 'a' to represent all three at once), and r, w, and x to represent permissions:
chmod u=rwx
chmod g+wx
chmod o-rwx
chmod a=rwx
The first command clears the old permissions for user and sets the user permissions to read/write/execute, with the permissions to group and others remaining the same. The second command doesn't erase the previous permissions, it simply adds write and execute permissions to group and keeps user and others the same. The third keeps user and group the same, and removes read/write/execute from others. The fourth erases the old permissions for user, group, and others and set all three to read/write/execute. If you don't use the first bit (u, g, o, or a) and simply do something such as "chmod +x /some/file", it will assume that you're talking about the user permission; it's good to specify user anyway if you're working with system files just to make it easier to find and correct mistakes.
For blacklisting, I know there's fail2ban, which works with iptables, but I'm not sure if it will work for what you're asking for. From what I understand, fail2ban watches for a DDoS attempt and does a temporary ban of the ip via iptables.
I'm definitely tweaking and changing the permissions, but also trying to find something akin to (your most excellent suggestion) fail2ban.
I think I'm going to have to get into the guts of iptables and see how I can implement it in a manner which is proactively blacklisting unwelcome IPs from writing to the server.
Thanks again for your continued guidance in this.
@Emufarmers:
The problem is that you're running vulnerable software. Adjusting permissions isn't going to fix that, and neither is fail2ban.
Note that ghumayun didn't say what software or what version (other than to say he's using WordPress), so we don't know if anything is actually out of sate. I already included making sure everything was up-to-date in my suggestions. If he already did that, or if he was already using the latest software when the files got uploaded, then making sure permissions and iptables are set correctly will help enhance security further.
@fhumayun:
@Emufarmers Mightily unhelpful comment. Please consider yourself exempted from making any more.
The only way to exempt Emufarmers from making comments is for a staffer to ban him, but as a moderator on another forum, I can say it's highly unlikely that'll happen. He was offering a suggestion without actually reading the entire thread. That could be due to him being tired, or simply a lack of attention. The staffers are more likely to watch you for a short bit. I doubt they'll actually ban you, but your response can be perceived as being rude. Note my response – I pointed out that you didn't mention any software versions, so we can't say if your software is outdated, and then I pointed out that I already suggested updating software, and in doing so I tried to avoid sounding rude.
The fact remains that there will always be ways in which services can and will get compromised. The spirit of my request was to enjoin into dialog where others in the linode community have been through these type of situations. In addition, I started this thread to demonstrate (to Linode staffers) that I am legitimately concerned about managing client sites while not having my account held in violation of any TOS.
That being said, my message to Emufarmers was simply to point out that I considered his response to be unhelpful at best – offering absolutely nothing to the discussion -- and condescending at worst. I can and will ignore the chap if he decides to continue trolling.
@fhumayun:
That being said, my message to Emufarmers was simply to point out that I considered his response to be unhelpful at best – offering absolutely nothing to the discussion -- and condescending at worst. I can and will ignore the chap if he decides to continue trolling.
Having a computer (regardless of if it's a server, a personal computer, etc.) compromised is reason for concern of anybody who uses that computer. That's why we have forums such as this – so we can help each other with our issues.
Even if the comment was unhelpful, you don't need to be rude about it. You asked for help, we are trying to help. Really the only reason his suggestion was unhelpful was because I had already suggested it when I first replied to your thread. He may have missed it purely by mistake, in which case there would be no reason to get mad. Of course, we can't tell if he did miss it by mistake or if he's actually trolling, and to not offer the benefit of the doubt will only get you in trouble. And if it was condescending, it would have been toward me since I was the one who suggested permissions and fali2ban, and he immediately discounted my suggestion. However, I did not make a rude comment toward him saying his suggestion was unhelpful, I politely explained why it was unhelpful.
If you already checked all your software to see if it was up-to-date, it would be better to say that than to make it look like you have the authority to exempt someone from making comments. If you don't think someone's comments are worth looking at, you are completely free to ignore them, just quote whomever you are responding to so we don't get confused. If someone does become a problem, you can report that person to a moderator or admin. Generally the admins and moderators are indicated by having their names in a different color than everybody else's. The admins and moderators on this forum will have "Linode Staff" under their user names. Unfortunately, I can't find any kind of "Report" link or form of some such, so at best you can find a Linode Staff member who's been active on the forum recently and click the "pm" button to send them a private message about the problem user.
(I know phpBB 3.x has a report form, this would be better because all the admins/moderators would see the reports rather than just a single staffer, so the abusive user can be taken care of much faster; and since we're on the topic of security, phpBB 2.x doesn't receive updates any more
I'd suggest you step away from the computer for awhile and calm down. If you allow your frustration to get in the way, you'll be seen as the troll. Having someone break into your computer is very frustrating. I had it happen to me quite a few times and was a big panic because I had some sensitive data on my hard disk, and I wasn't able to figure out what to do to secure my computer until I was able to calm down and listen. If you allow yourself to become too frustrated, you'll end up overreacting to everyone who tries to help, and that will only cause the Linode staffers to ban you from the forums (and IRC if you start getting mad at people there).
* I don't think adjusting permissions would have helped in this instance. The cache directory needs to be writable by the webserver user, and it sounds like that's the user that did the uploads. One thing that would help mitigate this sort of attack would be to configure Apache so that it won't execute PHP and .htaccess files in writeable directories. There's <url url="https://www.mediawiki.org/wiki/Manual:Security#Upload_security">~~[](https://www.mediawiki.org/wiki/Manual:Security#Upload_security)~~[a page on MediaWiki.org](https://www.mediawiki.org/wiki/Manual:Security#Upload_security)</url> that has information on how to do this.
fhumayun: I apologize for coming off as rude. I assure you that my advice was well-meaning, though I agree I could have offered it in a nicer fashion.</r>
AllowOverride Nonedirectories
@Emufarmers - I appreciate the gesture and thank you for the very helpful information about prevention of script execution in writable directories. I'll investigate it.
I should say, this is only the first time with this client hosted over 3 years that I've had a problem. I'd rather explore the elimination of other options before I lose a valued client – certainly don't want to throw the baby out with the dirty bath water.
My intention is to have processes in place which harden the linode, as well as monitor against repeated exploits of the same type. If there is enough statistical occurrence of vulnerability, and band-aiding doesn't cut it, it might warrant the disabling of certain features and even risk losing the client if necessary. But, I'd like the client to feel they're working with me to keep their content hosted rather than to feel threatened by a policy of intolerance. This was exactly why I shifted to Linode VPS from reselling Shared Hosting.
As for the permissions, I definitely wanted to make sure that the client hadn't created world-writable directories where it wasn't necessary. Covering some basics is always helpful, even if it's not the specific solution to this particular issue. In fact, this issue inspired me to write up a quick (cron) shell script that checks against a whitelist of known directories which require world-write and recursively resets any others back to proper perms. A counter is kept and an alert is generated if permissions for the same directories continuously needing to be reset.
@Vance - Thank you very much indeed for this pointer! I'm looking into it.
Webmin based IP Tables GUI ConfigServer Services
DenyHosts (auto-ban SSH attacks)
Fail2ban (auto-ban web/mail bots)
PortSentry (works with IPTables)
Tripwire (ubuntu) - baseline CRC file checking
htaccess hardening
PHP Hardening
wordpress firewalling
auto-ban PHP script (works with htaccess)
exploit scanner
"World-writable" doesn't mean that anyone in the world can write to that directory. Setting permissions to 777 doesn't make the directory a public bulletin board for the entire world. The "world" in "world-writable" means "any user of this computer". In order for a remote attacker to write something to a "world-writable" directory, he first needs to gain access to a program that is already running as one or another user of your Linode. Typically, this is a PHP script with an unfixed security vulnerability. If an attacker knows how to exploit that script, he will be able to do whatever the script's functionality and permissions allow it to do. If you don't have any vulnerable scripts, a permission of 777 is harmless (although it's still not a good idea).
WordPress is a security minefield, even with the latest version. The core itself has historically contained a very large number of vulnerabilities. Even worse, when such vulnerabilities are found, they are often patched in a haphazard manner instead of fixing the underlying problem (such as not using prepared statements for DB queries, or not using a proper HTML sanitizing function).
Plugins have an even worse track record, because 1) they're developed and maintained by third-parties who often don't know enough about security; 2) plugin code gets much less critical scrutiny than WordPress itself; 3) bugs in plugins are often not fixed quickly enough; 4) even when bugs are fixed, users often don't update plugins as frequently as they need to; and 5) plugins often try to bypass security measures built into WordPress, exacerbating all of the above problems. As for themes, they are a disaster the scale of Chernobyl. A significant percentage of free WordPress themes found on the Internet contain obfuscated code with advertising and other shady functions, which may also contain unknown vulnerabilities.
So keep every plugin and theme up to date, and remove anything you don't really really need. That little plugin that gives you some innocuous-looking eye candies? Dump it. That theme you tried out for 5 minutes a couple of years ago? Dump it. Also, when you remove plugins and themes, don't just disable their functionality. Delete them without a trace. Nuke them from orbit. It's the only way to be sure.
Never, ever trust WordPress to keep your server safe.