Tips on tracking down malicious proxy activity

@Yardboy:

This is where I think I have identified the evidence of some sort of proxied activity - I do not host any of these sites in the GET traffic. But I do not know what to do from here to track down the issue.
Note that in all of these cases your server is generating a 302 redirect in response to these queries, which seems odd if you don't in fact host any of those sites. Those ought to be some sort of 4xx failure.

You gave a lot of information but left out your actual Linode name/address (the two local addresses I see in your netstat don't seem to respond on port 80), so I can't see what the redirection is for, but that's the first place I'd start. Generate some requests to your server with bogus host names and see what responses you are getting back.

In terms of your configuration, verify that you are explicitly defining the virtual hosts that you intend to implement on your machine, and if you want to have a default block, have it just reject any requests.

– David

Hi David -

Thanks for your input. Sorry about that, at that point in the afternoon I had actually turned off the linode and started rebuilding a new one from scratch, concerned that my server had been completely compromised. I completed that process, and then when I booted the apache service I immediately had the same traffic issues.

The server is currently at 66.228.60.174 and the site I have brought up is http://www.melray.com - a simple static one pager at this point. Yet, I continue to have massive traffic hitting the server. I do notice, however, that now the responses to the traffic are 404's, so I'm wondering if the 302 redirects were due to the proxying and/or the fact that it was a rails app on the backend.

Example

23.19.47.6 - - [15/Nov/2012:09:48:22 -0500] "GET http://ads.creafi-online-media.com/st?ad_type=iframe&ad_size=728x90§ion=3285715&pub_url=${PUB_URL} HTTP/1.0" 404 652 "http://www.ladykaleidoscope.com/index.php?option=com_content&view=article&id=2196:2012-06-18-22-12-02&catid=38:big-women-fashion&Itemid=57" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8b3) Gecko/20050712 Firefox/1.0+"
199.15.113.199 - - [15/Nov/2012:09:48:22 -0500] "GET http://adprudence.rotator.hadj7.adjuggler.net/servlet/ajrotator/206999/0/vj?z=adprudence&dim=145922&kw=&click=&abr=$scriptiniframe HTTP/1.0" 404 690 "http://weightlossdiets2011.com/lose-weight-fast-by-fast-weight-loss-diets-are-you-ready.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; Alexa Toolbar)"
173.234.145.116 - - [15/Nov/2012:09:48:22 -0500] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0§ion=2888726&banned_pop_types=29&pop_times=1&pop_frequency=0&pub_url=${PUB_URL} HTTP/1.0" 404 644 "http://qtsfinancial.com/index.php?option=com_content&view=category&layout=blog&id=43&Itemid=99&limitstart=25" "Opera/9.80 (Windows NT 5.2; U; en) Presto/2.6.30 Version/10.63"

I also now notice that my netstat on this new server has a lot of stuff in it, mostly SYN_RECV and ESTABLISHED entries.

Example

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 *:smtp                  *:*                     LISTEN      3683/master
tcp        0      0 localhost:953           *:*                     LISTEN      2478/named
tcp        0      0 *:7833                  *:*                     LISTEN      2322/sshd
tcp        0      0 *:imaps                 *:*                     LISTEN      2447/dovecot
tcp        0      0 *:pop3s                 *:*                     LISTEN      2447/dovecot
tcp        0      0 localhost:9000          *:*                     LISTEN      3580/php-fpm.conf)
tcp        0      0 localhost:10024         *:*                     LISTEN      2943/amavisd (maste
tcp        0      0 localhost:10025         *:*                     LISTEN      3683/master
tcp        0      0 localhost:mysql         *:*                     LISTEN      2464/mysqld
tcp        0      0 *:submission            *:*                     LISTEN      3683/master
tcp        0      0 *:pop3                  *:*                     LISTEN      2447/dovecot
tcp        0      0 *:imap2                 *:*                     LISTEN      2447/dovecot
tcp        0      0 li72-23.members.li:http kenyon.hottermelly:1166 SYN_RECV
tcp        0      0 li72-23.members.li:http 23.19.63.196.rdns.:3357 SYN_RECV
tcp        0      0 li72-23.members.li:http rdns.ubiquity.io:1029   SYN_RECV
tcp        0      0 li72-23.members.li:http simplypayment.net:1262  SYN_RECV
tcp        0      0 li72-23.members.li:http 199.192.152.40:1330     SYN_RECV
tcp        0      0 li72-23.members.li:http 23.19.63.234.rdns.:2098 SYN_RECV
tcp        0      0 li72-23.members.li:http 23.19.63.197.rdns.:4314 SYN_RECV
tcp        0      0 li72-23.members.li:http 108.62.75.131.rdns:3823 SYN_RECV
tcp        0      0 li72-23.members.li:http sent.selectiveinfo:4616 SYN_RECV
tcp        0      0 li72-23.members.li:http responsechat.net:2655   SYN_RECV

Looking up some of these domains they seem to be connected to DDoS attacks, but I'm not sure how this one little site might attract that kind of attention. Via iptables I only have 80, 443, 8080 and my non-standard ssh port open on this server. I've tried to add some syn-flood rules but not getting much improvement.

> Generate some requests to your server with bogus host names and see what responses you are getting back.

This is the kind of thing I'm getting off the new server:

$curl --resolve www.melray.com:ad.adseverplus.com http://www.melray.com/st

<title>404 Not Found</title>

# Not Found

The requested URL /st was not found on this server.

Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.

* * *

<address>Apache/2.2.22 (Ubuntu) Server at www.melray.com Port 80</address>

Is this all just referer spam traffic, then? The load is making my actual sites run incredibly slowly, how might I defend against this?

Thanks again.

@Yardboy:

The server is currently at 66.228.60.174 and the site I have brought up is http://www.melray.com - a simple static one pager at this point. Yet, I continue to have massive traffic hitting the server. I do notice, however, that now the responses to the traffic are 404's, so I'm wondering if the 302 redirects were due to the proxying and/or the fact that it was a rails app on the backend.
Hard to say just from the prior logs, since either Apache itself or your backend could have generated a redirect. I'm not positive, but I believe that when given an absolute URI in a request (such as in these cases), Apache ignores the host and serves content from the specified relative location on the default host. If that default host is proxying, then your back-end will see the request, and how it deals with it is entirely dependent on the back-end code. But for example, if the back-end then generates a redirect that it thinks is to a local resource, but includes the full original URI, it could be sending the client off-site, and essentially act as a fully open proxy.

The good news is that 404 at least means nothing untoward is happening and the requests are just being rejected. But if you go from the static site back to your old configuration it wouldn't surprise me if the old behavior would just resume.

It's not absolutely clear how (or if) the redirects were being abused without being able to see the actual value of the redirect, but at the least it's not desirable behavior to even answer such requests.

> Looking up some of these domains they seem to be connected to DDoS attacks, but I'm not sure how this one little site might attract that kind of attention. Via iptables I only have 80, 443, 8080 and my non-standard ssh port open on this server. I've tried to add some syn-flood rules but not getting much improvement.
Yeah, you'd be surprised - all it takes is one member of the right "community" to discover an open server (relay, proxy, etc…) and it'll spread like wildfire.

> This is the kind of thing I'm getting off the new server:

$curl --resolve www.melray.com:ad.adseverplus.com http://www.melray.com/st

<title>404 Not Found</title>

# Not Found

The requested URL /st was not found on this server.

Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.

* * *

<address>Apache/2.2.22 (Ubuntu) Server at www.melray.com Port 80</address>

Is this all just referer spam traffic, then? The load is making my actual sites run incredibly slowly, how might I defend against this?

Unfortunately, once you've been identified, and the traffic is reaching your server, there's not a lot you can do aside from trying to drop or discard it as quickly as possible with as little impact as you can. Hopefully over time the servers pointing at you will give up and move on to new unsecured targets.

The good news is that for this specific sort of traffic the requests themselves are small and easily identifiable (they are for hosts you don't serve) so it should be possible to get them out of the equation quickly and with minimal overhead.

In your sample query above you're still using your proper domain name (and just a known bad URL), so it's not quite the right test. But for example, here's what I get when I attempt a query to your server with a bogus host (foo.bar):

> telnet www.melray.com 80
Trying 66.228.60.174...
Connected to www.melray.com.
Escape character is '^]'.
GET / HTTP/1.0
Host: foo.bar

HTTP/1.1 200 OK
Date: Thu, 15 Nov 2012 20:31:07 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.18
X-Runtime: 145
ETag: "29992c07ace40e91173b94d8e3305b29"
Cache-Control: private, max-age=0, must-revalidate
Set-Cookie: _icc_session=5e4116bb0a004d2083ab5bf76240358b; path=/; HttpOnly
Content-Length: 3592
Status: 200
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=utf-8

(... page contents ...)

So why is your server answering a query for domain foo.bar? You want to lock that down since it still shows that you are proxying any random request to your rails back-end. I did try using an absolute URI and still got a 404 this time, so that would seem to be a little different than before.

Now, that could still be ok if your rails application knew exactly what domains to answer for, but clearly it's willing to answer anything at this point. So it's probably easiest to lock things down before that stage. Make sure your Apache configuration only accepts queries for the specific server host names that you support. I would suggest having nothing in your default vhost block except for a deny all for /.

Even rejecting the request will still tie up an Apache worker to service the continuing bogus requests, but it ought to be quick. However, if you find that still being a performance penalty you might consider putting something like nginx in front of Apache as it can absorb the bogus requests with less overhead (you could also use the custom 444 response configuration to immediately close the connection without response). But since at the moment you're still using rails to process the bad requests, the overhead is probably much higher than it needs to be and just having Apache drop it immediately might be fast enough.

Alternatively you could try an iptables based solution like log scanners such as fail2ban which would recognize the 404s and adjust iptable-level blocking for those addresses over time.

Oh, one caveat - this assumes the bad requests aren't supplying the absolute URI but still using your proper (melray.com) host beneath the covers. That's unlikely, but if so, then you might have to do more work than just vhost configuration to filter them out.

But by all means, the first step you need to do is lock down your configuration so you stop servicing requests for domains that aren't you.

– David

I really appreciate you taking the (significant) time to respond with all this great info. I will work on getting things locked down as you suggest.

I do notice that the traffic on melray.com has dropped off. I am instead seeing it now on one of my wordpress domains running under ispconfig, which is where the open vhost declarations are now. Will get that taken care of right away.

Thanks again.

yb.