Google bot spamming my site for admin links?
Today I am stunned to find my access log is 17mb (just for today! They usually run under 500kb) with traffic almost entirely form googlebot. The IP addresses listed actually are google IP's. The pages trying to be accessed are almost entirely from equine/admin/* type links which are no longer in use on my site so are generating 404's but would have been inaccessible previously (access denied). I checked webmaster tools and almost all the 404 errors to my site for the last couple of days are to these links and I mean PAGES of them.
So my question is why is google spamming my site trying to find links that aren't there? The changes I made to my sites structure have been in place since March so it's not like I changed things last week. I did add equine/admin as denied on my robots.txt but is there anything else I should do?
Thanks everyone!
1 Reply
google follows links. If spammy links have been posted to your site by a bot then there's nothing you can do about it. 2000 visitors a day is not high traffic and any linode should easily be able to cater for this.
Are you using a caching plugin or better yet APC/Memcached? if not, i'd advise getting this set up. You could always set up apache/nginx to block the /enquine/admin/* pages, this would reduce your load as requests aren't going back to your database. You could also block bots from accessing your site via apache/nginx config files but this is NOT recommended.