Was I hacked? Help please.

Those directory names look a little too obviously sketchy… I'd expect something more subtle, you know what I mean?

It might be worth checking with the ISPConfig folks to see what might have created those directories… it was something running as root, so it probably wasn't a simple PHP exploit (although you are running a likely-exploitable version of PHP).

Whenever you discover any suspicious files, it is a good idea to make a tarball of the suspicious files before deleting them. That way, if some sort of exploit is suspected, those files can help investigate. Run:

tar cjvf suspcious.tar.bz2 /path/to/suspicious/files

You can specify a directory file here. That will generate suspicious.tar.bz2 containing your files in whichever directory and place it in the directory you run that command from. After doing that, you can remove the files.

Also, remove that tarball from the machine you found them on. For example, you can move the tarball off of your Linode and onto your home computer, then from there burn the tarball to a CD or put it on a flash drive, then delete it from your home computer.

It is also good to check your logs – ssh logs, ftp logs (if you have a ftp server), and any logs pertaining to your web servers (apache, php, etc). If you have anything else on your Linode that is available to the outside world, examine those logs too. Look for anything that looks suspicious. Make a copy of any log that contains suspicious information so that suspicious info doesn't get overwritten during any further logging. Some programs will occasionally rename the log and start a new one, e.g. rename /var/log/example.log to /var/log/example.log.1 and create an empty /var/log/example.log, to make the logs smaller and easier to read, and to have the old logs available for situations like this, but it's a good idea to make a copy just in case.

@Piki:

It is also good to check your logs – ssh logs, ftp logs (if you have a ftp server), and any logs pertaining to your web servers (apache, php, etc). If you have anything else on your Linode that is available to the outside world, examine those logs too. Look for anything that looks suspicious. Make a copy of any log that contains suspicious information so that suspicious info doesn't get overwritten during any further logging. Some programs will occasionally rename the log and start a new one, e.g. rename /var/log/example.log to /var/log/example.log.1 and create an empty /var/log/example.log, to make the logs smaller and easier to read, and to have the old logs available for situations like this, but it's a good idea to make a copy just in case.
Of course, if your system was compromised, the first thing the attacker is going to do is erase anything suspicious from the logs.

It is possible for the attacker to forget to do this, especially a newbie attacker, or someone who recently started using Linux. Also, every distro is different, while most use /var/log for this, the exact layout for /var/log can be different, and it is possible for the user to move these logs via config files for their software. It is possible to have "separated logs" for things like apache, so I'd have my main apache log in /var/log (or wherever I decide to tell apache to hide it), and individual access and error logs for each of my vhosts, each hidden in other directories. With correct permissions on these logs and good security setup, even if my web sites allows people to upload stuff, the logs will still be well hidden. I can also make it very hard to access root – disable root logins and passwords in my ssh daemon, and force the use of RSA keys, and as long as my private key is passworded and I guard it carefully, the attacker would need to find another exploit, which could be made difficult with proper permissions and iptables, and with well-written php hosted on my server.

Nevertheless, it is good to check logs just in case, make the tarball of the suspicious files and move it off site, and to also check to see if your config files changed (which I forgot to mention in my previous post, config files could possible be changed to make future exploits easier).

Thanks everyone!

The key points are:
* I have deleted the files without archiving them first. Should they re-appear I will make sure to make a copy first.

• I went though the logs before posting here and did not find anything concerning (which made me even more suspicious). I will look again when I get to a place with access tonight.

• What should I do next? Would anyone has any advice on what steps to take. Should I do nothing? Should I take some pre-emptive action? Where did these weird directories with names that include "Trojan.Vundo" and "Worm.Palevo" come from?
  If people are not sure, can anyone recommend a forum where folks may know about this?

Thanks a lot!!

-Start fresh with a new linode

-Check, double check, triple check your sites for compromised code. I'd recommend running through an IDS of sorts in the future.

-NEVER, EVER, EVER DOWNGRADE–EVER. For any reason. If the application CANNOT WORK with upgraded software, ditch it for someone who keeps updated.

-Run something like Modsecurity

-For every site on your linode, change their database passwords, update the site configs

-For any 'users' of any forums or login systems on any of your sites, require them to change their passwords.

-Announce that you've found a compromise and are dealing with it and working to not have one again.

-Read up on websecurity. OWASP has great information on this.

Thanks A-KO.

I really appreciate your advise. However, it is not practical in some details:

* Some applications may require an older runtime version (PHP, Java, .NET, a library, whatever it is). I am very enthusiastic about running the most up-to-date version, but from the point of view of businesses that use applications, this is the reality.

• Rebuilding the server is very time consuming. Of course, I would do that if I knew what went wrong. But without understanding the problem, I might invest a lot of time just to run into trouble again. I would like to understand the issue first, before deciding whether it warrants taking the server down and how I can make sure it does not happen again.

Any ideas on how to investigate what is going on?

Cheers!

Easy: Out of date software. Typically if you're not applying security patches or the latest security releases then you're going to run into trouble.

There are plenty of ways to do web app penetration testing and I'm not really inclined to go into the massive detail in this thread right now, as it's not the time and place.

I would recommend running something like Nessus or Metasploit, or pay for a person or company to do it for you to find vulnerabilities and information disclosure in your system.

Back to the software point at hand, you quoted Microsoft software in a forum regarding Linux. Not that I'm getting on you for that, but the fact that the patch release strategy is typically entirely different.

For example, PHP.NET no longer supports PHP 5.2.x, which means all security flaws found in it will no longer be patched. Any new security flaws found will not be provided with upstream fixes. You're free to patch whatever you can, but you run that risk.

A business running this software will typically have an IDS or IPS which will detect attacks in real time. So they have an extra level of security that looks for common/used exploits like those found in Metasploit–realtime, as the network passes through (look at SNORT).

Again, you have to consider the release cycle of each piece of software you use, and certain distributions have different levels of support and release cycles, and they typically work around some of the upstream providers. You need to be very proactive.

Other things you can look at are MODSECURITY, a form of IDS/IPS that operates directly in Apache and helps guard against the most common of exploits and disclosures. For example, it will help guard against common SQL Injection attacks, information leakage, SQL queries that try to pick from the mysql.users table, amongst many other attempts at exploits. It's very difficult to work with unless you know REGEX really well.

Apache's default logs will not include POST requests, even on the most verbose settings--which is how many attacks come in. I typically use mod_security (though there are other logging mechanisms) that can do full logs of the entire request from start to finish. You see every bit of data that was sent and received from the server. The log files can climb fast, I typically push a few hundred MB of logs/month even on a small website. Thankfully, text compresses really well.

@gragus:

Some applications may require an older runtime version (PHP, Java, .NET, a library, whatever it is). I am very enthusiastic about running the most up-to-date version, but from the point of view of businesses that use applications, this is the reality.

Then run an older LTS-style release that has the right major revision of the software that you need with the latest backported security fixes. If you need software even older than this, you're asking for trouble, because you're going to be open to all sorts of security holes, and whatever you need to do is probably not going to work all that far into the future with dead-end software.

@gragus:

Rebuilding the server is very time consuming. Of course, I would do that if I knew what went wrong. But without understanding the problem, I might invest a lot of time just to run into trouble again. I would like to understand the issue first, before deciding whether it warrants taking the server down and how I can make sure it does not happen again.

"time consuming" should never come into the equation when figuring out if you've been badly compromised enough to warrant a rebuild. You either have, and must rebuild, or haven't and don't need to.

> "time consuming" should never come into the equation when figuring out if you've been badly compromised enough to warrant a rebuild. You either have, and must rebuild, or haven't and don't need to.

Very true. I am looking to figure out exactly what happens before deleting and re-building the server.

If you absolutely require PHP 5.2, Ubuntu 8.04 LTS has 5.2.4, with security fixes backported, and it will be supported for another 18 months or so. There is also Drupal 5.7 in hardy, but it is not in the main repository and may not get much support going forward.

From drupal.org,

> Therefore, people adopting Drupal for their web or CMS project should plan for periodic upgrades of their project to the latest major release (every 12-24 months) …

This pretty much means that you are going to have to, at least once a year, do whatever is necessary to migrate to a new major release of Drupal, or you're going to have Problems. If you haven't done that this year, you're probably having Problems. It's like replacing furnace filters or changing engine oil.

Lucky my furnace hasn't caught fire… :P Though it does break down twice a year with the change in seasons, which can and will happen on a server of any kind without proper maintenance. Hands down and fists banging on the table, keeping up-to-date is one of the three most important things you can do to reduce your chances of compromise -- if there is a newer version of your software that contains a security fix (and there undoubtedly will be), you should definitely update regardless of what it takes.

Another important part of securing your server's security is checking over your server's firewall. The best firewall policy is to allow only what you absolutely need to and to block everything else. If it doesn't need to be accessible to the outside world, don't allow it (e.g. if you store your site database, such as a MySQL or SQLite database, on the same server everything that uses that database, you don't need to allow it to the outside world since your site doesn't need to go through the firewall to access a local database).

The third part of the process is to make sure that your server uses secure passwords, and that these passwords are NOT sent across the network in clear text (i.e. make sure they are encrypted when they are being sent across the Internet). There are plenty of posts on this forum for securing your ssh access. Common methods include changing the ssh port if you're allowing passwords to confuse the bots (attackers can easily work around this with a port scan) or to require a private and public key for ssh. You can also combine both methods (change the port AND require ssh keys), though again, a port scan can be used to work around that.

Aside from this, you should check over your system's configuration with a secure system in mind. Even a slight configuration error in anything with network access can potentially (but not always) open you up to attack. It is also good to log into the admin section of any web applications you run (e.g. forums, blogs, CMS's, etc.) and check over the configuration in those to see if there are potential security hazards.

It is possible that these files were uploaded through a vulnerability of Drupal and/or PHP which could mean no other part of the system was compromised except dirs writable by the webserver or more precisely the PHP process, which might explain why you don't see anything suspicious in other system logs.

Seems to me that some of the pages in your Drupal system(s) got hijacked to include links to these files which seem to be drive-by download malware*. Perhaps search the content tables to see if they appear in nodes, although this is not 100% as the filenames could be encoded in the compromised pages. Also check templates.

This also means that rebuilding the node might not solve the problem completely if the problem is confined to the content in the database.

*) Google search "Trojan.Vundo" and "Worm.Palevo".

Simply rebuilding won't resolve the issue, the whole point of rebuilding is to erase anything else that may have been compromised. It's just not practical to look through every single file on a system to find out what else (if anything) was compromised when you could just look through the essentials to find what might have allowed the compromise, erase and rebuild, then use what you learned to prevent future attacks. Combine that with our other suggestions, and any server should be pretty safe. Of course, absolute security is impossible, but that's why software developers keep pushing out security updates (even for Drupal and PHP), and also why when we are compromised, we should do a comprehensive search through our systems (not a complete search) and find out what we can do to improve.

Lesson to use an intrusion detection system like Aide or Tripwire. Personally I use Aide, and I also keep it's db checksum independently (the script mails it to me on each build) so I can check if Aide itself has been compromised.

This alone can prevent such problems in the future as the system will (should, at least) show all the files that have changed and should not have. Naturally, those files that SHOULD change, like logs etc…, shouldn't be covered by Aide. But at least it can cover the most important ones like /bin, /sbin, /usr/sbin, /etc and perhaps some files in /var (like package manager's). Update Aide DB on each system update, program (de)installation or config change.